Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JBR-4578 Add Java ATK Wrapper #475

Open
wants to merge 38 commits into
base: jbr21
Choose a base branch
from
Open

Conversation

tanya011
Copy link

@tanya011 tanya011 commented Nov 6, 2024

  • java-atk-wrapper used to be part of OpenJDK (JDK-8204862, IJPL-136231), but it was removed.
  • java-atk-wrapper is essential for accessibility in Java applications, including IntelliJ IDEA: IJPL-58438, JBR-4578.
  • AccessibleAnnouncerTest.java test is passed.
  • java-atk-wrapper has an LGPL 2.1 license, which can be up-cast to GPL-2.0, allowing the source code to be used.

@OnePatchGuy OnePatchGuy self-requested a review November 6, 2024 13:45
@tanya011 tanya011 force-pushed the jbr21 branch 3 times, most recently from 4af7c31 to 081ca50 Compare November 19, 2024 16:56
@vprovodin
Copy link
Collaborator

All changes in JBR should be accompanied with tickets in YT related to the JBR project. Ticket-id should be placed at the beginning of the commit message. Within this JBR ticket we may make proper changes in docker files. Please note the libraries you aded to docker files actually are not available for Oracle Linux 8 that is build platform for jbr21.

@tanya011 tanya011 changed the title IJPL-58438 Add Java ATK Wrapper JBR-4578 Add Java ATK Wrapper Nov 28, 2024
Library to enable accessibility on gnome linux.
Copy link
Member

@OnePatchGuy OnePatchGuy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the patch! It's been a hard work, hasn't it? :)

I've done reviewing only the changes in the build scripts (everything under the make directory). All the questions and suggestions are below. Reviewing the rest is coming.

From now on, please don't rewrite the commits history so that I'll be able to observe only the changed parts in newly added commits (we'll later squash them during merging).

Comment on lines 123 to 127
NEEDS_LIB_ATK=true
NEEDS_LIB_AT_SPI2_ATK=true
NEEDS_LIB_GLIB=true
NEEDS_LIB_GOBJECT=true
NEEDS_LIB_GLIBCONFIG=true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The patch adds new build-time dependencies, but:

  1. Our CI doesn't respect them (yet), thus if we merge the patch as is, it'll immediately get broken. E.g. what I get if I try to make a build in a docker container as described in the README:

Package atspi-2 was not found in the pkg-config search path.
Perhaps you should add the directory containing 'atspi-2.pc'
to the PKG_CONFIG_PATH environment variable
Package 'atspi-2', required by 'virtual:world', not found
Creating support/modules_libs/jdk.accessibility/libatk-wrapper.so from 16 file(s)
/JetBrainsRuntime/src/jdk.accessibility/linux/native/libatk-wrapper/AtkWrapper.c:25:10: fatal error: glib.h: No such > file or directory
#include <glib.h>
^~~~~~~~
compilation terminated.

(I get the same on Ubuntu 22 before installing the dependencies).

I can suggest 2 following ways of solving the problem:

  • to make the corresponding changes to our build scripts, docker images, etc. and include them into this PR ;
  • to make the changes provided in this patch disabled by default (i.e. apply --disable-java-atk-wrapper by default) and deal with all this CI stuff later after merging the patch .

Anyway, please note @vprovodin noticed earlier that Oracle Linux 8 which we're using as a build platform (via a docker image) doesn't provide these new dependencies, so it may become a serious obstacle for this patch.

  1. The build instructions in README haven't been updated correspondingly.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build instructions in README haven't been updated correspondingly.

I updated README, but not sure where to locate build instructions

Copy link
Member

@OnePatchGuy OnePatchGuy Jan 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure where to locate build instructions

Our README is currently describing 2 ways of how to build a JBR for Linux:

  • Using a specially designed docker container - the chapter Linux (Docker)
  • Using an Ubuntu as a host system - the chapter Ubuntu Linux

For now there's probably nothing to change in the Docker chapter, but Ubuntu chapter should be supplemented with the new packages and instructions how to enable/disable ATK wrapper and how to pass custom paths to it. Examples could be Windows's chapters Enable optional NVDA screen reader support and Disable optional JAWS screen reader support.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done at f83061c

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.
Let's file an issue about fixing CI and enabling back the patch by default and we can close this thread :).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note: here seems the full list of Ubuntu 22 packages that must be additionally installed to build the patch:

  • libatspi2.0-dev
  • libatk1.0-dev
  • libglib2.0-dev
  • libatk-bridge2.0-dev

-DATSPI_MINOR_VERSION=$(shell pkg-config --modversion atspi-2 | cut -d. -f2) \
-DATSPI_MICRO_VERSION=$(shell pkg-config --modversion atspi-2 | cut -d. -f3), \
LDFLAGS := $(LDFLAGS_JDKLIB), \
LIBS_linux := -ljvm $(ATK_LIBS) $(AT_SPI2_ATK_LIBS) $(GLIB_LIBS) $(GOBJECT_LIBS), \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding -ljava and/or -lawt as well if needed (I don't know).

@OnePatchGuy
Copy link
Member

OnePatchGuy commented Jan 15, 2025

My main question about the entire patch is the following: as you've written in the description, this functionality was removed some time ago due to a lot of bugs mentioned in e.g. IJPL-136231:

The problem is that the tool breaks the Swing threading rule and causes lots of exceptions & UI artefacts

and in JBR-4578:

Important! ATKWrapper contains bugs that can crash the JBR.

Also known issues:

  • Lists with complex renders work slowly;
  • Tree nodes are not voiced;
  • comboboxes are not spoken.

All these problems must be solved directly in ATKWrapper.

By putting all this logic back, do you state that all those problems have been fixed and now AWT wrapper is in production-ready quality?

UPD: still actual crash https://discourse.gnome.org/t/sigsegv-in-g-type-check-instance-is-fundamentally-a-0x11/24285

Copy link
Member

@OnePatchGuy OnePatchGuy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I seem to have done reviewing everything related to the Wrapper initialization.
Besides that, this "round" also contains a few comments about the resource management and some general notes.

Comment on lines 741 to 770
if (toolkit.getSystemEventQueue() != null) {
toolkit.getSystemEventQueue().push(new EventQueue() {
boolean previousPressConsumed = false;

public void dispatchEvent(AWTEvent e) {
if (e instanceof KeyEvent) {
if (e.getID() == KeyEvent.KEY_PRESSED) {
boolean isConsumed = AtkWrapper.dispatchKeyEvent(new AtkKeyEvent((KeyEvent)e));
if (isConsumed) {
previousPressConsumed = true;
return;
}
} else if (e.getID() == KeyEvent.KEY_TYPED) {
if (previousPressConsumed) {
return;
}
} else if (e.getID() == KeyEvent.KEY_RELEASED) {
boolean isConsumed = AtkWrapper.dispatchKeyEvent(new AtkKeyEvent((KeyEvent)e));

previousPressConsumed = false;
if (isConsumed) {
return;
}
}
}

super.dispatchEvent(e);
}
});
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a real need to intercept all the key events? All the keyboard input gets dependent on third-party components' logic we don't control. In the worst cases we may be getting EDT freezes and the inability to use keyboards at all.

Also, I'm not certain how exactly the queues pushing/poping works, but this approach most likely displaces the IntelliJ's own EventQueue installed at https://github.com/JetBrains/intellij-community/blob/794ebfdcb71781022a1357533be9f5fc352408b0/platform/platform-impl/src/com/intellij/ide/IdeEventQueue.kt#L139 .

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is at least one key event that we need to dispatch: IJPL-164705, so we cannot remove it completely.

Yes, Intellij has its own EventQueue and to fix the issue above, we would like to add a new dispatcher using the JBR API: https://code.jetbrains.team/p/ij/repositories/ultimate/revision/a6886190ed6084964ab4b31f01b03b19ccf9c21d. (I will create a new pull request for the JBR API after this one is approved.) It is assumed that it will be execute the code from dispatchEvent(AWTEvent e) function.

This approach does not replace IntelliJ's queue, quite the opposite, but I think there is no guarantee that it will not. Can we remove the push of a new EventQueue in AtkWrapper() and delegate this responsibility to the swing project?

Copy link
Member

@OnePatchGuy OnePatchGuy Feb 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is at least one key event that we need to dispatch: IJPL-164705

Could you please elaborate which one exactly? It's not clear from the issue description.
Also, if there's only a specific set of keystrokes, let's restrict the logic only to it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At first, I thought the issue was only with pressing the right and left arrow keys, but on Ubuntu 22.04, orca doesn't announce any key events : caret movement, text typing, pressing delete, tab, etc. So, we need to process all key events.
I cannot reproduce this problem on Ubuntu 24.04. If we do not process key events, everything is announced. The difference is in the version of orca.

Also, is it true that we should not consume any events?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See 0fd66b9

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I right that the new logic sends all key events both to ATK and to AWT/Swing? Doesn't it mean we can just replace installing of new EventQueue with a new global event listener, just how it's done for window, focus and container events?
Also let me know if some of your questions above still require answers.

@OnePatchGuy
Copy link
Member

BTW, I think each file that gets updated should also have its copyright header updated. One of the most recent examples of our header can be found e.g. in https://github.com/JetBrains/JetBrainsRuntime/blob/f89a6626d56dad71c3429c662bdad89f6b9a0591/src/java.desktop/unix/classes/sun/awt/wl/WLDisplay.java, but I don't know how to properly merge the existing headers with our one.

1. loadAtkBridge() in static block

2. Changes in initNativeLibrary:
- return False instead of exit(1)
- throw an exception if initNativeLibrary returns false
1. event doesn't get freed
2. synchronization
/usr/local/include/glib-2.0/gobject/gtype.h:2732: note: this is the location of the previous definition
 2732 | #define GTYPE_TO_POINTER(t) ((gpointer) (guintptr) (t)) GOBJECT_AVAILABLE_MACRO_IN_2_80
1. return boolean to throw an exception if something is wrong
2. unref jni_main_loop and jni_main_context if the thread is null
3. undo the effect of jaw_accessibility_init performed a few lines above
4. call initNativeLibrary() and loadAtkBridge() in initAtk()
@tanya011 tanya011 requested a review from OnePatchGuy February 11, 2025 08:29
Copy link
Member

@OnePatchGuy OnePatchGuy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have finished reviewing the rest of the initial code. Have found a lot of issues, some of them are critical memory leaks or, in opposite, use-after-free errors

else if (javaKeyLocation == KeyEvent.KEY_LOCATION_RIGHT)
javaKeyCode += RIGHT_OFFSET;

if ((gdkKeyInfo = (GNOMEKeyInfo) keyMap.get(Integer.valueOf(javaKeyCode))) != null) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keyMap only holds GNOMEKeyInfo values, so let's make its type more specific and remove the downcast from here and everywhere else (if any).

private static void initializeMap() {
keyMap = new HashMap<>(146); // Currently only 110, so allocate 110 / 0.75

keyMap.put(Integer.valueOf(KeyEvent.VK_COLON), new GNOMEKeyInfo(0x20a1, "ColonSign")); // GDK_ColonSign
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[refactoring] There are lots of meaningless integer boxing in this file, let's get rid of them.

Comment on lines +156 to +172
public static final class GNOMEKeyInfo {
private int gdkKeyCode;
private String gdkKeyString;

public GNOMEKeyInfo(int code, String string) {
gdkKeyCode = code;
gdkKeyString = string;
}

public int getGdkKeyCode() {
return gdkKeyCode;
}

public String getGdkKeyString() {
return gdkKeyString;
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[refactoring]

Suggested change
public static final class GNOMEKeyInfo {
private int gdkKeyCode;
private String gdkKeyString;
public GNOMEKeyInfo(int code, String string) {
gdkKeyCode = code;
gdkKeyString = string;
}
public int getGdkKeyCode() {
return gdkKeyCode;
}
public String getGdkKeyString() {
return gdkKeyString;
}
}
record GNOMEKeyInfo(int gdkKeyCode, String gdkKeyString) {}

Comment on lines +137 to +143
// Yes, this is crude, but Java does not provide another way.
String s = e.paramString();
int begin = s.lastIndexOf("rawCode=") + 8;
int end = s.indexOf(',', begin);
String rawcode_s = s.substring(begin, end);

keycode = Integer.valueOf(rawcode_s);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code may end up with exceptions or just give the wrong values e.g. if rawCode or a comma after it are absent.

I propose to extend sun.awt.AWTAccessor.KeyEventAccessor with a getRawCode method and then just:

Suggested change
// Yes, this is crude, but Java does not provide another way.
String s = e.paramString();
int begin = s.lastIndexOf("rawCode=") + 8;
int end = s.indexOf(',', begin);
String rawcode_s = s.substring(begin, end);
keycode = Integer.valueOf(rawcode_s);
keycode = AWTAccessor.getKeyEventAccessor().getRawCode(e);

break;
}
default: {
char[] chars = new char[1];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use String.valueOf(char) instead

* even been sent!
*/
static pthread_mutex_t jaw_vdc_dup_mutex = PTHREAD_MUTEX_INITIALIZER;
static jobject jaw_vdc_last_ac = NULL;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are the usages of this variable implemented correctly?

  1. Comparisons to other JNI references are performed through "raw" pointer comparisons, although different JNI references may reference the same Java object.
  2. It gets assigned to local JNI reference which are only valid during a JNI downcall.

Comment on lines +191 to +215
enum _SignalType {
Sig_Text_Caret_Moved = 0,
Sig_Text_Property_Changed_Insert = 1,
Sig_Text_Property_Changed_Delete = 2,
Sig_Text_Property_Changed_Replace = 3,
Sig_Object_Children_Changed_Add = 4,
Sig_Object_Children_Changed_Remove = 5,
Sig_Object_Active_Descendant_Changed = 6,
Sig_Object_Selection_Changed = 7,
Sig_Object_Visible_Data_Changed = 8,
Sig_Object_Property_Change_Accessible_Actions = 9,
Sig_Object_Property_Change_Accessible_Value = 10,
Sig_Object_Property_Change_Accessible_Description = 11,
Sig_Object_Property_Change_Accessible_Name = 12,
Sig_Object_Property_Change_Accessible_Hypertext_Offset = 13,
Sig_Object_Property_Change_Accessible_Table_Caption = 14,
Sig_Object_Property_Change_Accessible_Table_Summary = 15,
Sig_Object_Property_Change_Accessible_Table_Column_Header = 16,
Sig_Object_Property_Change_Accessible_Table_Column_Description = 17,
Sig_Object_Property_Change_Accessible_Table_Row_Header = 18,
Sig_Object_Property_Change_Accessible_Table_Row_Description = 19,
Sig_Table_Model_Changed = 20,
Sig_Text_Property_Changed = 21
};

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This enum is just a copy of AtkSignal.java. If you mark the constants from there with @Native annotation, they can be reused directly in native code (a native header org_GNOME_Accessibility_AtkSignal.h will be generated), thus avoiding the duplications.

Please also check all other native and Java enums for duplicates.

free_callback_para(para);
return;
}
para->child_impl = child_impl;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we increase the reference counter of child_impl here and then decrease during the cleanup of para? AFAIU the current code can lead to use-after-free errors.

Please revise all other usages of jaw_impl_get_instance for ownership sharing errors.

jniEnv, (*jniEnv)->GetObjectArrayElement(jniEnv, args, 1));
gchar* insert_text = get_string_value(
jniEnv, (*jniEnv)->GetObjectArrayElement(jniEnv, args, 2));
g_signal_emit_by_name(atk_obj, "text_insert", insert_position,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will g_signal_emit_by_name free insert_text? I suspect there are memory leaks here and in the similar case branches below.

} else if (type == type_released) {
event->type = ATK_KEY_EVENT_RELEASE;
} else {
queue_free_callback_para_event(para);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

event is leaking here.

Copy link
Member

@OnePatchGuy OnePatchGuy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have reviewed the recent updates.

private static final Object lock = new Object();
private static AtkWrapperDisposer disposerInstance;

static {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's convert the class to a lazy initialized singleton, so that threads will be created only when required and not when the class is getting loaded.

Comment on lines +74 to +77
AtkWrapper.releaseNativeResources(nativeReference);
obj.clear();
obj = null;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can these lines be executed out of the synchronized block?

import sun.awt.util.ThreadGroupUtils;

@SuppressWarnings("removal")
public class AtkWrapperDisposer implements Runnable {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's document the class, like what and how and why it does.

@SuppressWarnings("removal")
public class AtkWrapperDisposer implements Runnable {
private static final ReferenceQueue<Object> queue = new ReferenceQueue<>();
private static final Hashtable<java.lang.ref.Reference<Object>, DisposerRecord> records =
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused field.

Comment on lines +124 to +125
exports sun.java2d to
jdk.accessibility;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this required?


if (!jaw_log_file) {
perror("Error opening log file " JAW_LOG_FILE2);
return FALSE;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For jbooleans, the proper values JNI_TRUE and JNI_FALSE. The same for Java_org_GNOME_Accessibility_AtkWrapper_loadAtkBridge

JAW_DEBUG_I("Thread create failed: %s !", err->message);
g_main_loop_unref (jni_main_loop);
#if ATSPI_CHECK_VERSION(2, 33, 1)
g_main_context_unref(jni_main_context);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure, but shouldn't we do smth like atk_bridge_set_event_context(NULL) here as well?

Comment on lines 741 to 770
if (toolkit.getSystemEventQueue() != null) {
toolkit.getSystemEventQueue().push(new EventQueue() {
boolean previousPressConsumed = false;

public void dispatchEvent(AWTEvent e) {
if (e instanceof KeyEvent) {
if (e.getID() == KeyEvent.KEY_PRESSED) {
boolean isConsumed = AtkWrapper.dispatchKeyEvent(new AtkKeyEvent((KeyEvent)e));
if (isConsumed) {
previousPressConsumed = true;
return;
}
} else if (e.getID() == KeyEvent.KEY_TYPED) {
if (previousPressConsumed) {
return;
}
} else if (e.getID() == KeyEvent.KEY_RELEASED) {
boolean isConsumed = AtkWrapper.dispatchKeyEvent(new AtkKeyEvent((KeyEvent)e));

previousPressConsumed = false;
if (isConsumed) {
return;
}
}
}

super.dispatchEvent(e);
}
});
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I right that the new logic sends all key events both to ATK and to AWT/Swing? Doesn't it mean we can just replace installing of new EventQueue with a new global event listener, just how it's done for window, focus and container events?
Also let me know if some of your questions above still require answers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants