So, are you using a modbus protocol over wifi/tcp or serial or what ?
It does seem that enabling the modbus at least aggravates your problem so I personally would look deeply into that area. Is it the protocol formatter on the esp8266 causing problems, or what/how it communicates over a given GPIO or wi-fi etc? Is it serial/uart attached? Is the uart/wifi/? Getting swamped, running out of buffers/memory etc?
Perhaps you can describe it here, with as much detail as possible, as although I don't know modbus, many different I/O problems stem from the same thing and one of us may spot a problem.