Multithreaded TCP server with a thread pool

A thousand clients should not cost eight gigabytes of stack. A fixed worker pool behind a mutex-guarded queue keeps resource usage flat no matter how many connections arrive.

Built: July 2025
Language: C++17
Dependencies: none - stdlib + POSIX
Environment: Ubuntu 22.04, g++ 11
Workers: fixed pool, configurable

Source on GitHub

Accept loop, bounded queue, fixed workers - and what the two memory strategies actually cost.

01The naive server, and why it collapses

Nearly every introductory TCP server looks like this, and it is correct right up until it is used:

while (true) { int c = accept(...); std::thread(handle, c).detach(); }

The bug is not in the code, it is in the resource model. A thread carries roughly 8 MB of stack reservation, so a thousand concurrent clients ask the kernel for about eight gigabytes before a single byte is processed. Long before you reach that, the scheduler is spending more time switching between threads than running them, and there is no ceiling anywhere - the server's resource usage is whatever clients decide it is.

A thread pool inverts that. A fixed set of workers drains a shared queue, so concurrency is decoupled from connection count. Throughput becomes flat and predictable, and the failure mode under overload is a growing queue - which you can measure and shed - rather than an out-of-memory kill.

02Both strategies, side by side

Connections arrive at a steady rate below. Switch the strategy and watch what changes - and more importantly, what does not.

Live: connections arriving at a fixed rate

Same arrival rate in both modes. Only the resource curve differs.

queued 0 threads 4 stack 32 MB served 0

accept() loopnever blocks on a client

queue empty

worker poolfixed at 4, whatever arrives

In pool mode the thread count is flat at four and the queue absorbs the burst. In thread-per-client mode nothing queues - because every arrival immediately becomes another thread, and the stack figure is the one that ends the process.

03The three components

task_queue.hpp

TaskQueue

A thread-safe FIFO between the acceptor and the workers. Every access is guarded by a std::mutex; workers block on a std::condition_variable rather than polling, so an idle server burns no CPU at all.

thread_pool.hpp

ThreadPool

Owns a fixed set of workers, configurable at construction. RAII throughout - the destructor joins every worker, on any exit path. Copy and move are deleted, because thread ownership must be unique.

server.hpp

TCPServer

socket(), bind(), listen(), then an accept loop that does nothing but accept and submit. Handles SIGINT and SIGTERM for clean shutdown.

The key invariant: the accept loop never handles a client itself. It accepts, submits, and is immediately back in accept(). The moment that loop does real work, connection setup latency becomes coupled to request processing time, and the pool stops helping.

04Decisions that are easy to get wrong

Decision	Reasoning
`std::condition_variable`	Workers sleep until there is work. Polling would burn a core per idle worker for nothing.
`std::optional` from `pop()`	Returning `nullopt` signals shutdown without exceptions and without a sentinel task that every worker has to recognise.
`notify_one()` on push	One task needs one worker. `notify_all()` here wakes every thread to have all but one lose the race - a textbook thundering herd.
`notify_all()` on shutdown	The opposite case: every worker must wake and observe the stop flag, or the destructor blocks forever in `join()`.
`std::atomic` counters	Diagnostics are read constantly and written rarely. Atomics keep the stats path lock-free so observing the server does not contend with running it.
`SO_REUSEADDR`	Without it, restarting during `TIME_WAIT` fails to bind for up to a minute - which turns every test cycle into a wait.
`emplace_back` for workers	`std::thread` is not copyable; the threads must be constructed in place in the vector.

The notify_one versus notify_all pair is my favourite thing in this codebase, because the correct choice is the opposite in each direction and both are one token long. Getting either backwards produces something that still passes a light test and then behaves badly under load or hangs on exit.

05Testing it honestly

A concurrency bug that only appears under load is not going to be found by connecting one client by hand, so the server ships with a Python stress client that opens many simultaneous connections and verifies every response.

Concurrent clients, verified responses. Not just "did it not crash" - each client checks it got its own reply back, which is what catches cross-talk between workers.
Shutdown under load. SIGINT while the queue is full is where RAII and notify_all() either hold or deadlock. Testing teardown only on an idle server proves nothing.
Restart immediately. Stop and start back to back, which is what SO_REUSEADDR exists for.
Queue depth as the load signal. Growing depth means arrival rate exceeds service rate - the number to act on well before the machine is in trouble.

Built in two days on WSL2 Ubuntu with g++ 11, zero external dependencies - the C++17 standard library and POSIX, nothing else. It is deliberately the Linux-side counterpart of the CAN work: the same questions about priority, starvation and bounded resources, asked in a place where the answers are mutex and condition_variable instead of dominant and recessive bits.