When I looked through the Wireshark logs, it showed that the esp8266 initiated the TCP connection close.
Below is the response from AWS. My esp8266 sends a FIN, ACK shortly after
Frame 33: 214 bytes on wire (1712 bits), 214 bytes captured (1712 bits) on interface 0
Ethernet II, Src: Netgear_4a:bb:7e (a0:63:91:4a:bb:7e), Dst: HonHaiPr_5b:f2:57 (90:48:9a:5b:f2:57)
Internet Protocol Version 4, Src: 35.166.227.117, Dst: 192.168.1.5
Transmission Control Protocol, Src Port: 8883, Dst Port: 50078, Seq: 2897, Ack: 85, Len: 148
[3 Reassembled TCP Segments (3044 bytes): #28(1448), #31(1448), #33(148)]
Secure Sockets Layer
TLSv1.2 Record Layer: Handshake Protocol: Multiple Handshake Messages
Content Type: Handshake (22)
Version: TLS 1.2 (0x0303)
Length: 3039
Handshake Protocol: Server Hello
Handshake Type: Server Hello (2)
Length: 77
Version: TLS 1.2 (0x0303)
Random
Session ID Length: 32
Session ID: 594dd75b48565f644050eb9f09aba8369de719ef4fe825a0...
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
Compression Method: null (0)
Extensions Length: 5
Extension: renegotiation_info
Handshake Protocol: Certificate
Handshake Type: Certificate (11)
Length: 2912
Certificates Length: 2909
Certificates (2909 bytes)
Certificate Length: 1563
Certificate: 30820617308204ffa003020102021064c0578b6277f27035... (id-at-commonName=*.iot.us-west-2.amazonaws.com,id-at-organizationName=Amazon.com, Inc.,id-at-localityName=Seattle,id-at-stateOrProvinceName=Washington,id-at-countryName=US)
Certificate Length: 1340
Certificate: 3082053830820420a0030201020210513fb9743870b73440... (id-at-commonName=Symantec Class 3 Secure Server CA - G4,id-at-organizationalUnitName=Symantec Trust Network,id-at-organizationName=Symantec Corporation,id-at-countryName=US)
Handshake Protocol: Certificate Request
Handshake Type: Certificate Request (13)
Length: 34
Certificate types count: 3
Certificate types (3 types)
Signature Hash Algorithms Length: 26
Signature Hash Algorithms (13 algorithms)
Distinguished Names Length: 0
Handshake Protocol: Server Hello Done
Handshake Type: Server Hello Done (14)
Length: 0