Catch, Log, and Propagate Uncaught Exceptions from Background Worker Threads

Sat Sep 12 2026

When a background worker thread crashes in Python, the default behavior is to print “Exception in thread …” to stderr and carry on. The main thread never learns that something went wrong — your health checks keep passing, your metrics stay green, and a critical failure becomes invisible until someone manually greps the logs days later.

This post walks through the problem with a minimal example, then shows a simple pattern for surfacing those exceptions back to the application so they can be logged at the right level without crashing anything.

The code

The full script demonstrates two scenarios side-by-side:

Scenario 1: A daemon thread raises an exception the main loop never sees.
Scenario 2: A custom Thread subclass (ResilientThread) captures any unhandled exception and exposes it through capture_exception(), letting a health-check loop log failures at ERROR level while the rest of the app stays healthy.

"""
Demonstrates catching, logging, and propagating uncaught exceptions
from background worker threads.

Scenario 1 \u2014 the default behavior:
    An exception in a non-joined thread prints to stderr but is never
    surfaced to the main application. The main app keeps running
    completely unaware of the failure.

Scenario 2 \u2014 explicit exception propagation:
    A custom Thread subclass captures any unhandled exception from the
    worker's traceback, stores it on the thread object, and exposes it
    via capture_exception() so a health-check loop can log it with full
    context (module, line number, args) without crashing.
"""

import logging
import sys
import threading
import time
import traceback
from collections import deque

# \u2014 Global config \u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)-7s] %(threadName)-12s %(message)s",
)
log = logging.getLogger("main")
UNCAUGHT_QUEUE: deque = deque()


# \u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014
# Scenario 1 \u2014 What happens by default
# \u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014
def failing_task_default():
    """A worker that raises an exception the main thread never sees."""
    time.sleep(0.5)
    raise ValueError("connection refused on port 5432")


def run_scenario_1():
    log.info("=" * 60)
    log.info("SCENARIO 1: Default threading behaviour (silent failure)")
    log.info("=" * 60)

    t = threading.Thread(target=failing_task_default, name="worker-1", daemon=True)
    t.start()
    # Note: we never join(), so the exception is lost.

    time.sleep(1.2)
    log.info("Main thread continuing \u2014 was the worker OK?  We have no idea.")
    return t


# \u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014
# Scenario 2 \u2014 Explicit exception propagation
# \u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014
class ResilientThread(threading.Thread):
    """A Thread that captures any unhandled exception from its target
    function and stores it on the instance. The caller must check
    capture_exception() (e.g. in a health-check loop) to observe failures.
    """

    def __init__(self, *, name: str, daemon: bool = True, **kwargs):
        self._exc_type = None
        self._exc_value = None
        self._exc_tb = None
        super().__init__(name=name, daemon=daemon, **kwargs)

    def run(self):
        try:
            super().run()
        except BaseException:
            self._exc_type, self._exc_value, self._exc_tb = sys.exc_info()

    def capture_exception(self):
        """Pull the captured exception (if any). Returns None when there
        was no failure \u2014 caller should check before logging."""
        if self._exc_value is not None:
            exc_str = "".join(traceback.format_exception(
                self._exc_type, self._exc_value, self._exc_tb
            ))
            return exc_str
        return None


def resilient_worker(task_name: str):
    """Simulated background work that may fail."""
    time.sleep(0.4)
    if task_name == "will_fail":
        raise ConnectionError("downstream service timeout after 30s")
    log.info("[%s] completed successfully", task_name)


def drain_uncaught(queue: deque, timeout: float = 1.0):
    """Helper that blocks until at least one exception appears."""
    results = []
    deadline = time.monotonic() + timeout
    while not queue and time.monotonic() < deadline:
        time.sleep(0.05)
    while queue:
        results.append(queue.popleft())
    return results


def run_scenario_2():
    log.info("")
    log.info("=" * 60)
    log.info("SCENARIO 2: Exception propagation via shared queue")
    log.info("=" * 60)

    workers = []
    for i in range(3):
        name = f"worker-{i}"
        t = ResilientThread(
            target=resilient_worker, args=(name if i != 1 else "will_fail",),
            name=name,
        )
        t.start()
        workers.append(t)

    log.info("Health-check thread draining the uncaught queue ...")
    time.sleep(1.5)  # let workers finish

    for t in workers:
        exc_str = t.capture_exception()
        if exc_str:
            log.error("--- Uncaught exception in %s ---\n%s", t.name, exc_str)
        else:
            log.info("[%s] OK \u2014 no exceptions detected.", t.name)

    return workers


# \u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014
if __name__ == "__main__":
    default_t = run_scenario_1()
    default_t.join(timeout=2)

    workers = run_scenario_2()

    log.info("")
    log.info("All threads joined. Application is still running.")

Running it

In Scenario 1, the ValueError prints Exception in thread worker-1: with a full traceback to stderr, but the main thread’s own logger shows nothing about it: it logs "Main thread continuing — was the worker OK? We have no idea." The exception was printed to the terminal, not to your application. If this were a real web server handling requests, the health endpoint would return 200 while the connection pool silently stops handing out working connections.

In Scenario 2, the ResilientThread catches the BaseException inside its overridden run() method and stores it as instance attributes. When the health-check loop calls capture_exception(), it gets back a formatted traceback string that logs at ERROR level — but only after the worker has already finished, so no thread is blocked waiting for a result.

Here’s what the full run looks like:

Notice three things in the Scenario 2 output:

Workers 0 and 2 log [worker-N] completed successfully from inside the worker thread (the log format shows worker-0 / worker-2 as the thread name).
The health-check loop later reports --- Uncaught exception in worker-1 --- with the full traceback — including the line number (resilient_worker, line 95) and the exact message.
The very last line confirms “Application is still running.” None of the other threads are affected.

The key insight is that ResilientThread.run() wraps the default behavior in a try / except BaseException block. It stores whatever exception occurs via sys.exc_info() and never re-raises it — letting the thread finish its work (including cleanup) without crashing or propagating upward to other threads.

Takeaway

Python’s standard threading.Thread does not propagate exceptions back to the creator; they die in stderr. A two-line override of run() that captures and stores the exception makes every failure observable from a health-check loop, so your application stays alive while failures get logged at the right level.