I did some further testing , especially for ruling out that the impendance mismatch of the SMA feedline influences the measured spectrum.
For that I first checked if the RF Power of the ESP is within the range written in the ESP8266EX Datasheet.
I used a Minicircuits ZX47-40-S+ Power Detector (I'm too cheap for the Power Detector Head for the SA...)
Thats the output voltage vs RF input Power diagram of the Minicircuits Powerdetector
And that is what I measure
The ESP Datasheet states a output Power of +15.5 to +17.5dBm @ 72Mbps , my AP says the ESP is associated with 65Mbps
So I get roundabout +16dBm out of the ESP , but one have also have to take the RSMA to SMA Adapters and RF Cabling into account what I used to connect the Powerdetector to the ESP (~ -0.5dB) .
Well within the values in the Datasheet
I also did a S22 (aka 2 Port Insertion Loss measurement) on the 50 Ohm line between the RSMA connector and the 5.6pf coupling cap (the part of the PCB where I have a missmatch due to 10mils vs 26mils)
Doesn't look soooo bad afterall