Add device orientation information to header information

[sbfkcel]

This is useful if need to draw the device appearance correctly.

Related Discussions

#5704
#3605
#925

[rom1v]

Hi,

Thanks for your work.

However, as I said in #5704 (comment), there is no really need to know the device orientation, since it can be inferred from the frame size in practice, and there are cases where the "orientation" to write is not clear.

You replied:

But in some cases, it would be good to have the correct device rotation direction. For example, it is necessary to correctly draw the appearance of a device

You can do this based on the actual frame size.

In theory, it could make sense to pass the device orientation (at least in some cases) in packets headers (even if it could be argued that it's overkill to pass it for each packet/frame).

But then, the orientation must exactly match the frame it describes, which is not guaranteed here. There is no mechanism to make sure that the frame produced by MediaCodec is in the current device orientation (as returned by Device.getCurrentRotation(this.displayId)), it may have changed, so a few frames may be tagged with an incorrect orientation. On the contrary, the frame size is guaranteed to be in sync.

[sbfkcel]

Thanks for reply

The size of the screen can only be used to determine whether it is displayed horizontally or vertically.

If it is just a pure screen display, then it is meaningless to obtain the device direction.

However, if you need to draw the hole position of the front camera in the screen normally, or if you need to draw the position of the device appearance button (power button) as discussed above, then you need an accurate device direction.

I used to monitor the change of the screen size, and when the size changes, I used ADB to obtain the screen direction, which will cause a very large delay.

From the current test, the implementation in the code is relatively ideal. The human eye cannot clearly distinguish that the obtained value is abnormal.

Or is it feasible for us to make this item an optional method?

Currently it is also sent as part of the metadata extension. It should not be considered redundant.

[rom1v]

However, if you need to draw the hole position of the front camera in the screen normally, or if you need to draw the position of the device appearance button (power button) as discussed above, then you need an accurate device direction.

OK.

What is the client on your screenshot? Is it a modified scrcpy client, or an alternative client you wrote?

The human eye cannot clearly distinguish that the obtained value is abnormal.

If the orientation is sent "inline", it must be in sync with the encoded frames. Btw, it could allow to print a better error message for #1645.

The current protocol has two "packet types":

• a codec packet; for video: codec id (u32), width (u32), height (u32)
• a media packet for each frame (with a 12 bytes header)

To support your use case, I suggest to add a new packet type:

• a codec packet, with only the codec id (u32, i.e. 4 bytes)
• a session packet for video (12 bytes): width (u32), height (u32), orientation (u3, there a 8 possible orientation with --capture-orientation), sent every time the encoding is restarted (e.g. on rotation)
• a media packet for each frame (with a 12 bytes header)

The session packet could be distinguished from the media packet with a single bit, and for simplicity both headers would have 12 bytes.

It would be sent when a new encoding session starts, with the values in sync to what is passed to the encoder.

[sbfkcel]

Yes, I wrote an alternative client(browser-based).

I will try to make some adjustments based on your suggestions and then submit a new implementation.