Writing a BitTorrent client from scratch

I wanted to know what actually happens when you open a torrent, so I wrote a client. The rule I set was no dependencies: if the standard library did not have it, I had to write it. That turned out to be the useful constraint, because it meant I could not skip the parts I did not understand yet.

The thing that got me through it was the unofficial spec on wiki.theory.org. It has the byte-level detail the official one leaves vague, and it was open in a tab the entire time I was writing this.

Bencode first, because nothing works without it

A .torrent file is bencode: four types. Integers are i42e, strings are length-prefixed as 4:spam, lists are l...e and dictionaries are d...e with sorted keys. The decoder took an evening.

I ended up writing it as a dispatcher. One function looks at the byte at position i and hands off based on what it finds, and lists and dictionaries call back into it for their contents, so nesting sorted itself out without me writing anything special for it.

What I did not expect was that every parse function needs to return two things: the value, and the index it stopped at. I wrote the first version returning just the value and could not work out why nested structures came out wrong. There is no way to know where one item ends without the parser telling you, and searching for the terminator does not work because e shows up inside nested values too.

The other thing that bit me is that bencode strings are bytes, not text. The pieces field is 20 byte SHA-1 digests glued together, and the moment anything in the pipeline treats it as UTF-8 it either throws or quietly mangles it. Everything stays as bytes, which is why the keys look like b'piece length' throughout.

Then I found out I needed an encoder too, which I had not expected. The info hash that identifies a torrent is the SHA-1 of the info dictionary as it appeared in the file. You cannot compute it from the decoded structure unless you can re-encode that structure byte for byte, sorted keys and all. Get one byte wrong and the tracker does not know what you are asking for. So the encoder exists entirely to reproduce a hash.

Getting a list of peers

The tracker is an HTTP GET with the info hash, my peer ID, a port and compact=1 in the query string. I generate the peer ID once at import, a short prefix plus random bytes to fill it to twenty.

The compact response is where I lost an hour. It is not a list of anything readable, it is one long byte string where every peer is exactly six bytes: four for the IPv4 address, two for the port, big endian. You slice it in sixes and unpack. If the length is not divisible by six something has gone wrong upstream and there is no point trying to parse it, so I check that first.

The wire protocol is the easy half

Talking to a peer is mechanical once you have the spec open. A TCP socket, then a 68 byte handshake: one byte holding the number 19, the literal string BitTorrent protocol, eight zero bytes reserved for extensions I do not implement, then the 20 byte info hash and the 20 byte peer ID. I have the builder assert that it came out to 68, because when I got it wrong the failure was just a peer hanging up on me with no explanation.

After that it is length-prefixed messages with a one byte ID: choke, unchoke, interested, have, bitfield, request, piece. I ask for 16 KiB blocks and stitch them into pieces.

The thing I had read about but never really internalised until it broke: TCP is a stream, not messages. A single recv can hand back half a message, or two of them stuck together. My first attempt assumed one recv meant one message and it fell apart the moment the network got busy. Now everything goes through a recv_exactly helper that loops until it has exactly the number of bytes asked for, and raises if the peer disconnects partway.

The bitfield took me a couple of tries too. Peers announce what they hold as one bit per piece, and the ordering within each byte is high bit first, so piece i lives in byte i // 8 at bit 7 - (i % 8). I had that shift the wrong way round initially, which meant asking peers for pieces they did not have. It does not error. It just quietly does not work, which took a while to notice.

Where the actual thinking went

The spec tells you how to talk to one peer. It does not tell you how to run thirty of them at once without them all downloading the same piece.

What I settled on is one shared structure holding three sets: not_started, in_progress and done, with a single lock around every operation. A worker asks for a piece index, which moves it from the first set to the second. If it finishes, the piece moves to done. If anything goes wrong, the worker calls give_back and the index returns to not_started for someone else to try.

That give_back is the piece of it I would defend. Three different failures route through it: the peer does not have the piece, the socket dies mid-transfer, or the hash does not match. In all three cases the work is not lost, it is just released. Without it, one bad peer takes a piece of the file down with it and the download never completes.

One global lock is not clever. With thirty threads and pieces taking real time to transfer, contention is nowhere near the bottleneck, and the alternative is a locking scheme I would have to reason about at 3am. Writing to the file uses a second lock, since every thread seeks into the same handle at its own offset.

Peers send you garbage

Every piece has a SHA-1 in the torrent file, and you check it. This is not a theoretical safeguard. Run it against a real swarm and you will see mismatches print out. A peer can be broken, malicious, or serving a different version of the file.

So a piece is only written to disk after the hash matches. On mismatch the buffer is thrown away and the index goes back to the queue. At the end the whole ISO gets a SHA-256 that I compared against the checksum Canonical publishes. That was the moment it stopped being an exercise.

What I did not build

Rarest-first piece selection is the obvious gap. Real clients prioritise pieces that few peers have, so nothing vanishes from the swarm. I tested on Ubuntu, where essentially every peer has every piece, so a rarity calculation would have been code with no observable effect. I left it out rather than write something I could not see working.

Also missing: asking the tracker for fresh peers as the initial batch dies, which is fine on a healthy swarm and would crawl on a weak one; proper choke handling, so a thread choked mid-download currently waits rather than backing off; and uploading, since this only pulls.

The thing I keep coming back to is how much of it is not in the spec. The spec is a wire format. Which piece to ask for, what to do when a peer lies to you, how to share work across threads without them colliding: those are all yours to decide, and they are the whole program.