Decode multiple messages from the same buffer

[shahn]

How can I decode multiple UPER-encoded messages from a buffer where they are stored directly after each other, without delimiter? decode just takes a buffer, but neither shrinks it when reading bytes nor returning how many bytes it read to parse the first message. Initially I thought the impl of Decode for Vec<T> where T: Decode would help me, but that assumes that the messages are stored in an asn1 sequence rather than immediately one after another. I didn't see any kind of API to read this efficiently, did I overlook that or could it be added like a decode_multiple (would require alloc of course) or something like that?

[XAMPPRocky]

Thank you for your issue! You need to capture the decoder, which has an input method, that returns the remaining unparsed input.

https://docs.rs/rasn/latest/rasn/uper/struct.Decoder.html#method.input

[shahn]

Thanks, that does seem to work - but it is a bit awkward to use, as my messages are byte-aligned and the padding prevents me from re-using the decoder:

let mut input = BitStr::from_slice(data);
while !input.is_empty() {
    let mut decoder: rasn::uper::Decoder<'_, 0, 0> = rasn::uper::Decoder::new(input, DecoderOptions::unaligned());
    let x = MyType::decode(&mut decoder);
    [do something with x]
    let remaining = decoder.input();
    let (_, remaining) = remaining.split_at(remaining.len() % 8);
    input = remaining;
}

Is there a nicer way to achieve what I am looking to do?

[Nicceboy]

I added new API for OER codec to solve the same issue:

rasn/src/coer.rs

Lines 18 to 22 in d97c44d

	pub fn decode_with_remainder<T: crate::Decode>(input: &[u8]) -> Result<(T, &[u8]), DecodeError> {
	let decoder = &mut Decoder::<0, 0>::new(input, de::DecoderOptions::coer());
	let decoded = T::decode(decoder)?;
	Ok((decoded, decoder.remaining()))
	}

Maybe there is a chance to add similar?

[Nicceboy]

This does not do any extra allocations. It just moves the reference of the original input slice.

[shahn]

Well, hrm. If the messages are truly one after the other, without padding, that function wouldn't work. I think maybe for UPER it would be better to have some kind of flag for the Decoder to indicate that the messages are byte-aligned one after another and then you can just call T::decode multiple times with the same decoder, same as it would work right now? Not quite sure

[XAMPPRocky]

Thank you for your issue!

I think how I would want to solve this issue is by providing an iterator struct that is a wrapper around decoder.

let iter = rasn::de::iter::<rasn::uper::Decoder>(input);

loop {
  while let Some(Ok(value)) = iter.next() {
    // handle value
  }

  iter.append_bytes(new_data);
}

[shahn]

Thank you for the reply! Do you expect a new message to be aligned at the next byte border then, or not? Right now the Decoder does not consume padding at the end of a message up to the next byte border (which is why I can't just read the next message using the same Decoder, which would also work fine). I am just not sure what the expected behaviour would be like, here.

[XAMPPRocky]

I believe all complete encodings are byte aligned no?

[shahn]

If that is the case, then maybe the problem is there's a bug in the UPER decoder? Because the input() method does not return an aligned BitStr, and if I try to just decode the next message, it fails

[XAMPPRocky]

It is likely that they we haven't set it so the UPER parses the padding at the end of the encoding.

[Nicceboy]

Yeah, encodings should be always byte aligned:

[shahn]

Basically, iiuic, you'd need an entry function that isn't T::decode for whole messages, because otherwise you won't know whether padding should be consumed. This is actually a bit tricky to fix due to the recursive structure of rasn.

[Nicceboy]

I can take a look for this over weekend. At least the decode_with_remainder should be easy to add since on that level we can deduce the byte alignment. Maybe I can look for the iterator approach too.