It turned out to be the USB to Serial adapter. I was using some of the really cheep FTDI ones (the ones with the red solder mask). I tried 3 different ones, all with the same result.
Finally ordered an adapter with an Si Labs chip and, bingo! Uploaded just fine. I have still had some failures but usually can get the firmware uploaded after a handful of attempts.
Hope this helps.