Missing received data with UsbSerialJtag

[yanshay]

I'm communicating over UsbSerialJtag, doing back and forth message exchange with a laptop.
Many times I see that the laptop sends data and I miss some of the data at the beginning of the message when the laptop responds to my message. The laptop start sending after it receives my packet so it's not full-duplex communication.
There is very minimal work between write and read. Only issue is that I'm debugging this so there are logs going out on the non Jtag serial, so maybe debugging slows it down.
As if form the time data is sent but I miss the start of it, around 4 bytes or so are missing. Most of the times it is missing, sometimes its not.
I didn't see any buffers configurable for UsbSerialJtag.
What could cause this and is there a way to avoid it?