I have been realising the very same recently. A few years ago I was running around 11 dead at 12 stone, last year at 13 stone my times were closer to 12.00 even though my 50m was sub 6 sec. Bare in mind I hadn't trained speed end. atall but my top speed just felt awkard.
One of my old training partners is age 42 and still running sub 11, sub 22 and sub 50. As well as being a machine he reckons it is down to his 69kg weight. He hasn't slowed much in 20 years and has weighed pretty much the same since then.
Although sprinters appear big and muscular on TV they are still very lean and light.