I just took a look at Linux's nf_conntrack_proto_udp.c to see what's happening. It looks like Linux starts the UDP timeout at 30 seconds, but extends it to 180 seconds when it sees traffic both ways, assuming it's "some kind of stream."