Module 11: Networks and the web

Applications define messages; transport protocols connect endpoints; IP routes packets between networks; link technologies move frames across a local hop. Each layer adds information needed for its responsibility. A socket endpoint includes an address and port: the address identifies a network destination, while the port selects a service context. A packet can cross several routers; its route is not the same as an application-level conversation. Loss, delay and reordering are normal conditions a protocol must account for.

DNS maps domain names to records, including addresses. A resolver can consult caches and query the distributed hierarchy. Cached records have lifetimes; changing a record does not instantly update every client. A hostname is not inherently one permanent IP address. The URL also includes a scheme, path and possibly a port or query. Our lab uses numeric loopback to isolate protocol behaviour from external DNS and routing; it does not demonstrate a full public DNS lookup.

TCP provides an ordered byte stream with transport-level reliability and flow control. It does not preserve application message boundaries: one send may require several reads, or several sends may arrive in one read. A zero-length recv result signals the peer has closed its sending side. UDP sends datagrams, retaining boundaries but not guaranteeing delivery or ordering. Choose around application requirements. A timeout indicates the caller did not complete in time, not that the peer definitely did no work.

A protocol can use delimiters, fixed widths or length prefixes. Our frame contains a four-byte unsigned big-endian length followed by that many UTF-8 bytes. Read the header exactly, validate the length against a maximum, then read the body exactly and decode it. Length counts bytes, not characters. The helper deliberately limits each recv to two bytes so partial reads are exercised. Reject truncated frames rather than silently treating partial content as a valid message.

HTTP expresses requests with methods, targets and headers, and responses with status codes, headers and optional bodies. GET retrieves a representation; POST requests resource-specific processing. 200 indicates success; 404 indicates no matching resource. JSON is one body format, independent of HTTP itself. HTTPS uses TLS to authenticate the server and protect traffic under its trust assumptions. Encryption does not validate application input or authorise a user. Retrying a timed-out mutation needs an idempotency design to avoid duplicates.

Download each script and run it in a terminal with Python 3.11 or later: python m11_stream.py. On Windows, py -3 is an alternative; on some systems use python3. The labs use only the standard library. Predict the result before running, then complete the variations. Run without -O so assertions remain enabled. Outputs below were captured by the builder. Code and output are identical in both language editions.

One finite server sends a frame and closes. The client assembles partial reads and checks a 1024-byte limit.

Download m11_stream.py

"""Receive a length-prefixed message even when reads return partial chunks."""
import socket
from threading import Thread
def read_exact(stream, count):
    result = bytearray()
    while len(result) < count:
        chunk = stream.recv(min(2, count - len(result)))
        if not chunk:
            raise EOFError("connection closed before complete frame")
        result.extend(chunk)
    return bytes(result)

with socket.socket() as listener:
    listener.bind(("127.0.0.1", 0)); listener.listen(1); listener.settimeout(5)
    payload = b"Dune,Foundation"
    errors = []
    def serve():
        try:
            connection, _ = listener.accept()
            with connection:
                connection.sendall(len(payload).to_bytes(4, "big") + payload)
        except Exception as error:
            errors.append(error)
    worker = Thread(target=serve); worker.start()
    with socket.create_connection(listener.getsockname(), timeout=5) as client:
        length = int.from_bytes(read_exact(client, 4), "big")
        if length > 1024: raise ValueError("frame too large")
        result = read_exact(client, length)
    worker.join(timeout=5)
    assert not worker.is_alive() and not errors and result == payload
    print("frame bytes:", length, "decoded:", result.decode("utf-8"))

frame bytes: 15 decoded: Dune,Foundation

1. Use UTF-8 text containing 中 and recompute byte length.
2. Advertise a longer body than is sent.
3. Advertise length 5000 and verify early rejection.

Inspect the JSON body, content type and status. The server binds only locally and shuts down after the checks; http.server is a learning tool rather than a production server.

Download m11_http.py

"""A finite localhost HTTP experiment with JSON, statuses and cleanup."""
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Thread
from urllib.error import HTTPError
from urllib.request import urlopen
class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/books":
            self.send_error(404); return
        body = json.dumps([{"id": 1, "title": "Dune"}]).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "application/json; charset=utf-8")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers(); self.wfile.write(body)
    def log_message(self, *args):
        pass

server = HTTPServer(("127.0.0.1", 0), Handler)
thread = Thread(target=server.serve_forever); thread.start()
try:
    base = f"http://127.0.0.1:{server.server_port}"
    with urlopen(base + "/books", timeout=5) as response:
        books = json.load(response)
        print("GET /books:", response.status, books)
        assert response.status == 200 and books[0]["title"] == "Dune"
    try:
        urlopen(base + "/missing", timeout=5)
    except HTTPError as error:
        print("GET /missing:", error.code); assert error.code == 404; error.close()
    else:
        raise AssertionError("missing resource must be 404")
finally:
    server.shutdown(); server.server_close(); thread.join(timeout=5)
    assert not thread.is_alive()

GET /books: 200 [{'id': 1, 'title': 'Dune'}]
GET /missing: 404

1. Add /health returning a small JSON object.
2. Return an empty list for an empty catalogue.
3. Explain why missing paths differ from zero matching books.

Try before opening the solution. ★ applies an idea; ★★ combines ideas; ★★★ asks for design or proof.

Choose an answer for feedback; reset to retry. A text answer key is available without JavaScript.

• Python socket HOWTO — Read send/recv and message framing; explain partial reads.
• HTTP semantics RFC 9110 — Read GET and status code definitions, not the entire specification.

Trace a byte frame and an HTTP response separately. Explain what a timeout does not tell you. Module 12 persists catalogue state in a relational database.