The system sent data from an embedded target to a Windows PC via Ethernet / UDP / IP. The UDP protocol is defined as "unreliable" - packets of data may be lost, duplicated, or arrive out-of-order. Initially, we assumed that this was only an issue out in the Internet, where packets can travel vast distances by multiple routes and networks. We assumed that on a one-meter cable, there was nowhere for packets to "get lost". However, on rare occasions (one packet every few gigabytes) the Windows stack would misplace a packet.
When we realised this, I modified our software to handle missing / duplicated / out-of-order packets.
This taught me not to make assumptions about the reliability of other parts of the system and to build in robustness from the start.