If you check the D1 circuit, there's an ME6211 LDO supplied by the 5V pin that outputs up to 500mA on the 3V3 line - this is used to power the ESP and also for any external sensors that need power. The dropout voltage on the LDO is about 300-500mV so you can supply as low as 3.6-3.8V on the 5V pin and it will still work.
What Voltage is your LiPo charger circuit supplying? Can it be hooked up to the 5V pin? As it is the diode on your circuit is going to protect the LiPo so put one on the 5V side as well and you should be fine