The Python Developer's Guide to APIs: Requests Library Deep Dive

The Python Developer's Guide to APIs: Requests Library Deep Dive

Every modern Python project calls an API. Weather data, user authentication, payment processing, social media integration — it's all HTTP requests under the hood.

The requests library makes this clean and intuitive. But there's a big gap between the basic requests.get(url) tutorial example and the production-ready API client code you need for real projects.

This guide closes that gap.

Installation and Basic Usage

pip install requests

import requests

response = requests.get("https://api.github.com/users/gvanrossum")
print(response.status_code)  # 200
print(response.json())        # Python dict from JSON response

The response object has everything you need:

response.status_code        # 200, 404, 500, etc.
response.text               # Raw text response
response.json()             # Parse JSON → Python dict/list
response.headers            # Response headers dict
response.url                # Final URL (after redirects)
response.content            # Raw bytes (for images/files)
response.elapsed            # Time taken for the request

The HTTP Methods

# GET — retrieve data
response = requests.get("https://api.example.com/users")

# POST — create data
response = requests.post("https://api.example.com/users",
    json={"name": "Alice", "email": "alice@example.com"})

# PUT — replace data entirely
response = requests.put("https://api.example.com/users/1",
    json={"name": "Alice Updated", "email": "alice@example.com"})

# PATCH — update specific fields
response = requests.patch("https://api.example.com/users/1",
    json={"name": "Alice Updated"})

# DELETE — remove data
response = requests.delete("https://api.example.com/users/1")

Sending Data

Query Parameters (URL Parameters)

# Manual: requests.get("https://api.example.com/search?q=python&limit=10")
# Clean way:
params = {"q": "python", "limit": 10, "sort": "stars"}
response = requests.get("https://api.github.com/search/repositories",
    params=params)
print(response.url)  # Shows the full URL with encoded params

JSON Body (Most Common for POST/PUT)

data = {
    "title": "Learn Python",
    "completed": False,
    "priority": "high"
}
response = requests.post("https://jsonplaceholder.typicode.com/todos",
    json=data)          # Automatically serializes to JSON
                        # Sets Content-Type: application/json

Form Data

# For HTML form submissions (Content-Type: application/x-www-form-urlencoded)
response = requests.post("https://example.com/login",
    data={"username": "alice", "password": "secret"})

File Upload

with open("report.pdf", "rb") as f:
    response = requests.post("https://api.example.com/upload",
        files={"file": ("report.pdf", f, "application/pdf")},
        headers={"Authorization": "Bearer YOUR_TOKEN"})

Authentication

API Key in Header

import os

API_KEY = os.environ["API_KEY"]  # Never hardcode!

headers = {"Authorization": f"Bearer {API_KEY}"}
response = requests.get("https://api.example.com/data", headers=headers)

API Key as Query Parameter

response = requests.get("https://api.weathermap.org/data",
    params={"appid": API_KEY, "q": "London"})

HTTP Basic Auth

from requests.auth import HTTPBasicAuth

response = requests.get("https://api.example.com/private",
    auth=HTTPBasicAuth("username", "password"))
# Shorthand:
response = requests.get(url, auth=("username", "password"))

Error Handling — Production Patterns

import requests
from requests.exceptions import HTTPError, ConnectionError, Timeout, RequestException

def api_get(url: str, **kwargs) -> dict:
    try:
        response = requests.get(url, timeout=10, **kwargs)
        response.raise_for_status()  # Raises HTTPError for 4xx/5xx
        return response.json()
    
    except HTTPError as e:
        status = e.response.status_code
        if status == 404:
            raise ValueError(f"Resource not found: {url}")
        elif status == 401:
            raise PermissionError("Authentication failed")
        elif status == 429:
            raise RuntimeError("Rate limit exceeded — slow down requests")
        else:
            raise RuntimeError(f"HTTP error {status}: {e}")
    
    except ConnectionError:
        raise RuntimeError(f"Cannot connect to {url}")
    
    except Timeout:
        raise RuntimeError(f"Request timed out after 10s: {url}")
    
    except RequestException as e:
        raise RuntimeError(f"Request failed: {e}")

Always set timeout. Without it, a hanging server will hang your program indefinitely.

Sessions — Efficient Multi-Request Patterns

A Session persists headers, auth, and cookies across requests and reuses TCP connections:

import requests

class GitHubClient:
    BASE_URL = "https://api.github.com"
    
    def __init__(self, token: str):
        self.session = requests.Session()
        self.session.headers.update({
            "Authorization": f"Bearer {token}",
            "Accept": "application/vnd.github+json",
            "X-GitHub-Api-Version": "2022-11-28",
        })
    
    def get_user(self, username: str) -> dict:
        response = self.session.get(f"{self.BASE_URL}/users/{username}",
            timeout=10)
        response.raise_for_status()
        return response.json()
    
    def get_repos(self, username: str) -> list:
        response = self.session.get(f"{self.BASE_URL}/users/{username}/repos",
            params={"per_page": 100, "sort": "updated"},
            timeout=10)
        response.raise_for_status()
        return response.json()
    
    def close(self):
        self.session.close()


# Usage
client = GitHubClient(token=os.environ["GITHUB_TOKEN"])
user = client.get_user("gvanrossum")
repos = client.get_repos("gvanrossum")
print(f"{user['name']} has {len(repos)} repos")
client.close()

Retry Logic

For production API clients, add automatic retries with exponential backoff:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def create_session_with_retries() -> requests.Session:
    session = requests.Session()
    
    retry_strategy = Retry(
        total=3,
        backoff_factor=1,               # 1s, 2s, 4s between retries
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["GET", "POST"],
    )
    
    adapter = HTTPAdapter(max_retries=retry_strategy)
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    
    return session

session = create_session_with_retries()
response = session.get("https://api.example.com/data", timeout=10)

Pagination Patterns

Most APIs paginate large result sets. Common patterns:

Offset Pagination

def get_all_users(base_url: str, headers: dict) -> list[dict]:
    all_users = []
    page = 1
    
    while True:
        response = requests.get(base_url,
            params={"page": page, "per_page": 100},
            headers=headers,
            timeout=10)
        response.raise_for_status()
        
        data = response.json()
        if not data:
            break
        
        all_users.extend(data)
        page += 1
    
    return all_users

Cursor Pagination

def get_all_records(url: str, headers: dict) -> list[dict]:
    records = []
    next_url = url
    
    while next_url:
        response = requests.get(next_url, headers=headers, timeout=10)
        response.raise_for_status()
        data = response.json()
        
        records.extend(data["results"])
        next_url = data.get("next")  # None when no more pages
    
    return records

Calling the Claude API

A real example calling the Anthropic API with requests:

import os
import requests

def ask_claude(prompt: str, model: str = "claude-sonnet-4-6") -> str:
    response = requests.post(
        "https://api.anthropic.com/v1/messages",
        headers={
            "x-api-key": os.environ["ANTHROPIC_API_KEY"],
            "anthropic-version": "2023-06-01",
            "content-type": "application/json",
        },
        json={
            "model": model,
            "max_tokens": 1024,
            "messages": [{"role": "user", "content": prompt}]
        },
        timeout=30
    )
    response.raise_for_status()
    return response.json()["content"][0]["text"]

answer = ask_claude("Explain Python decorators in 3 sentences")
print(answer)

Frequently Asked Questions

What is the requests library?

Python's most popular HTTP library. Makes GET, POST, PUT, DELETE requests with a clean API, handling JSON, auth, and sessions automatically.

How do I handle API authentication?

Bearer token in Authorization header for most modern APIs. Store keys in environment variables, never in code.

How do I handle errors?

Use response.raise_for_status() + try/except for RequestException. Always set timeout parameter.

What is requests.Session?

Persists headers, auth, and cookies across requests. More efficient for multiple calls to the same API.

Final Thoughts

The gap between requests.get(url) and a production API client is mostly about error handling, retries, and sessions. The three patterns in this guide (proper error handling, session with default headers, retry adapter) cover what most real projects need.

For calling APIs in automation scripts, see our Python automation scripts guide. For building APIs that other programs call (instead of calling them), our FastAPI tutorial is the natural next step. And for the async version of HTTP requests when you need concurrency, our Python async/await tutorial covers aiohttp and httpx.