While the higher processor speed can be helpful it isn't the reason for the faster transfers. I think there are a few areas of improvement, some I understand and others I have not looked into. If I put the car into drive and it moves forward I don't have to know everything involved to make it do that. There are more details at https://github.com/esp8266/Arduino/issues/1853 and https://github.com/esp8266/Arduino/issues/1430. I now using the current git version but before I did that I had edited a buffer size (double) in one file and it gave a huge increase in speed.
A webserver is a file server. Just a more specialized one. It gets a file request and sends out that file. Maybe that first file is an HTML document and it lists a bunch more files for the client will then request. The problem with the ESP8266 is the limited connections it can handle. If you sent a request for half a dozen files all at once it will process the requests it can handle and then after that it seems to choke. So for my tests above I used a simple html page that included one CSS file to call and then the jpg image file.
But if I were to just put the jpg image file in the browser bar I could load just the one file.
I have been using the SPIFFS for all the files. I have not done any transfers with a SD card yet. I hope to soon though. Just too much to do. First I would do a web server but after that I hope to include ftp support.
While I consider the transfer rate usable for some web page stuff I wonder how useful it can be for transferring large image files. I guess it depends on how many you have and the file size.