core: Key input overhaul #19616

kjarosh · 2025-02-24T11:05:39Z

The main goal of this refactor is to unify key input across all platforms and move all Flash Player related behaviors to core in order to get rid of differences across platforms and minimize bugs.

Instead of providing KeyCodes in PlayerEvents, frontends now provide KeyDescriptors, which describe the pressed key on a high level. Then, Ruffle core does all necessary mappings to KeyCodes (and key location).

Additionally, this PR implements support for KeyboardEvent.keyLocation.

Fixes Problem with Key.getCode on AZERTY keyboard: the layout is always QWERTY on Ruffle #7381
Fixes [Cable Capers 2] Keyboard differences vs Flash version #1046

CC @evilpie

evilpie · 2025-03-01T14:42:29Z

desktop/src/util.rs

-        return Some(KeyCode::from_code(char.to_ascii_uppercase() as u32));
+fn alpha_to_logical(ch: &str) -> LogicalKey {
+    // TODO What if we get multiple chars?
+    if let Some(ch) = ch.chars().last() {


Didn't we use the first character before?

We weren't consistent:

https://github.com/ruffle-rs/ruffle/pull/19616/files#diff-7be54fe026a0ed3dc2fc1fe118b9028a8c46085c0e5df2a54087572ebbfa9666L219

https://github.com/ruffle-rs/ruffle/pull/19616/files#diff-62c679f6eccebf339181274aa5cd20101388c20fca85d34ccf5ef7e9c016cb77L163

I think getting the last char makes more sense in case of dead keys until we start supporting them properly.

web/src/input.rs

evilpie · 2025-03-01T15:00:49Z

web/src/input.rs

-        "F23" => KeyCode::F23,
-        "F24" => KeyCode::F24,
-        _ => KeyCode::UNKNOWN,
+        "Backquote" => PhysicalKey::Backquote,


Just curious: I tried to figure out where this order comes from and I think it's based on the code in e.g. https://developer.mozilla.org/en-US/docs/Web/API/UI_Events/Keyboard_event_code_values#code_values_on_windows. So why is the first not Escape?

Oh I had no reason to use Windows-specific docs :D The order is based on the spec: https://w3c.github.io/uievents-code/#code-value-tables

First it's categorized by section, then by the key's physical location, the order should be the same as in the PhysicalKey enum (you can see the sections there).

web/src/input.rs

evilpie · 2025-03-01T15:11:44Z

It is probably interesting to mention that this adds not yet used support for mapping physical keys to Flash's virtual key code.

core/src/input.rs

evilpie

Great work.

I feel like If we we also used winit as an abstraction for key handling on the web we could probably unify a lot of the code between desktop and web.

kjarosh · 2025-03-01T16:01:33Z

I feel like If we we also used winit as an abstraction for key handling on the web we could probably unify a lot of the code between desktop and web.

That is very true. Web and winit have compatible interfaces (winit basically uses the web interface in Rust), but as long as we don't use winit on web, we have to duplicate this interface :/

The main goal of this refactor is to unify key input across all platforms and move all Flash Player related behaviors to core in order to get rid of differences across platforms and minimize bugs. Instead of providing KeyCodes in PlayerEvents, frontends now provide KeyDescriptors, which describe the pressed key on a high level. Then, Ruffle core does all necessary mappings to KeyCodes (and key location in the future). This also adds (not yet used) physical key mapping in Ruffle.

This test verifies key input mapping on an 80% keyboard.

This test verifies key input mapping on a numpad.

This test verifies the behavior of keyLocation in key events.

torokati44 · 2025-03-03T20:42:09Z

core/src/events.rs

+#[derive(Debug, Clone, Copy)]
+pub struct KeyDescriptor {
+    pub physical_key: PhysicalKey,
+    pub logical_key: LogicalKey,
+    pub key_location: KeyLocation,
+}


Any suggestions on how to jam the contents of this neat struct (especially LogicalKey) all the way across this pipeline? 😵‍💫

https://github.com/ruffle-rs/ruffle-android/blob/main/app/src/main/res/layout/keyboard.xml#L121-L127
https://github.com/ruffle-rs/ruffle-android/blob/main/app/src/main/java/rs/ruffle/PlayerActivity.kt#L179-L188
https://github.com/ruffle-rs/ruffle-android/blob/main/app/src/main/java/rs/ruffle/PlayerActivity.kt#L97-L98
https://github.com/ruffle-rs/ruffle-android/blob/main/src/lib.rs#L487-L523
https://github.com/ruffle-rs/ruffle-android/blob/main/src/lib.rs#L400-L421

Or maybe I should leave all the XML/Kotlin parts operating on "softkeyboard scancodes" of a made-up layout, and translate it all in the Rust part? But that creates an uncomfy tie (or gap, depending on how you look at it) between how and where the layout is defined (including the button text), and what it "means" for the Ruffle player...

IMO when it comes to a simple, custom keyboard, you should use an approach similar to one used in tests, where each key has an identifier and is mapped into PhysicalKey/LogicalKey/KeyLocation. For more advanced keyboards I would expect all 3 fields to come from the virtual keyboard.

Maybe we can explore the idea of using the system keyboard on Android?

Maybe we can explore the idea of using the system keyboard on Android?

I don't think we can get separate down/up events from that, nor can it handle pressing multiple keys at the same time...

For pure text entry, sure, it would be great, but not for playing games, and such.

(The same mechanisms could/should trigger hiding our handmade keyboard and bringing up the system one as currently used for the same purpose on web IMO.)

Bur yeah, making each key have a proper identifier in its tag attribute, and carrying that string all the way into Rust, then dispatching on it there (turning it into a KeyIdentifier) sounds doable. But passing strings through the JNI FFI boundary is just so much more inconvenient than simple scalar values IIRC... 😩 Sure it's not impossible, just less nice...

You can always map it in Java and pass numerical values down. Not sure if that would require code duplication or not

Yeah but that would just require more messing around with encoding different (sometimes optional) info about all the keys into numbers and then those into Rust enumerants... This way it would be in one place at least.

Well, it would still require separating the methods to pass in the key events coming from our handmade virtual keyboard (with the string identifiers), and those coming from an actual physical keyboard connected say via Bluetooth (or USB or adb/scrcpy or whatever), in the form of an Android KeyEvent. Oh well, this isn't a big deal either I guess...

kjarosh added A-input Area: Input handling T-compat Type: Compatibility with Flash Player input Issues relating to user input in Flash content waiting-on-review Waiting on review from a Ruffle team member labels Feb 24, 2025

evilpie reviewed Mar 1, 2025

View reviewed changes

kjarosh force-pushed the key-code-mapping branch from f7d47e2 to fa35648 Compare March 1, 2025 14:51

evilpie reviewed Mar 1, 2025

View reviewed changes

web/src/input.rs Show resolved Hide resolved

evilpie reviewed Mar 1, 2025

View reviewed changes

web/src/input.rs Outdated Show resolved Hide resolved

evilpie reviewed Mar 1, 2025

View reviewed changes

core/src/input.rs Show resolved Hide resolved

evilpie approved these changes Mar 1, 2025

View reviewed changes

kjarosh force-pushed the key-code-mapping branch 2 times, most recently from 01e5c5e to c16dea4 Compare March 2, 2025 19:57

kjarosh removed the waiting-on-review Waiting on review from a Ruffle team member label Mar 2, 2025

kjarosh force-pushed the key-code-mapping branch from c16dea4 to 2a15cdf Compare March 2, 2025 21:37

kjarosh added 5 commits March 2, 2025 22:37

tests: Add avm2/key_input_80percent test

5bd2fcf

This test verifies key input mapping on an 80% keyboard.

tests: Add avm2/key_input_numpad test

474883f

This test verifies key input mapping on a numpad.

avm2: Add support for keyLocation in KeyboardEvent

207915d

tests: Add avm2/key_input_location test

b593080

This test verifies the behavior of keyLocation in key events.

kjarosh force-pushed the key-code-mapping branch from 2a15cdf to b593080 Compare March 2, 2025 21:38

kjarosh merged commit 9891184 into ruffle-rs:master Mar 2, 2025
22 checks passed

kjarosh deleted the key-code-mapping branch March 2, 2025 22:11

n0samu added the newsworthy label Mar 3, 2025

torokati44 reviewed Mar 3, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core: Key input overhaul #19616

core: Key input overhaul #19616

kjarosh commented Feb 24, 2025

evilpie Mar 1, 2025

kjarosh Mar 1, 2025

evilpie Mar 1, 2025

kjarosh Mar 1, 2025

evilpie commented Mar 1, 2025

evilpie left a comment

kjarosh commented Mar 1, 2025

torokati44 Mar 3, 2025

kjarosh Mar 3, 2025

torokati44 Mar 3, 2025

torokati44 Mar 3, 2025

torokati44 Mar 3, 2025

torokati44 Mar 3, 2025 •

edited

Loading

kjarosh Mar 3, 2025

torokati44 Mar 3, 2025

torokati44 Mar 3, 2025

core: Key input overhaul #19616

core: Key input overhaul #19616

Conversation

kjarosh commented Feb 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evilpie commented Mar 1, 2025

evilpie left a comment

Choose a reason for hiding this comment

kjarosh commented Mar 1, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

torokati44 Mar 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

torokati44 Mar 3, 2025 •

edited

Loading