-
Notifications
You must be signed in to change notification settings - Fork 209
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
JBR-4578 Add Java ATK Wrapper #475
base: jbr21
Are you sure you want to change the base?
Conversation
4af7c31
to
081ca50
Compare
All changes in JBR should be accompanied with tickets in YT related to the JBR project. Ticket-id should be placed at the beginning of the commit message. Within this JBR ticket we may make proper changes in docker files. Please note the libraries you aded to docker files actually are not available for Oracle Linux 8 that is build platform for jbr21. |
Library to enable accessibility on gnome linux.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the patch! It's been a hard work, hasn't it? :)
I've done reviewing only the changes in the build scripts (everything under the make
directory). All the questions and suggestions are below. Reviewing the rest is coming.
From now on, please don't rewrite the commits history so that I'll be able to observe only the changed parts in newly added commits (we'll later squash them during merging).
make/autoconf/libraries.m4
Outdated
NEEDS_LIB_ATK=true | ||
NEEDS_LIB_AT_SPI2_ATK=true | ||
NEEDS_LIB_GLIB=true | ||
NEEDS_LIB_GOBJECT=true | ||
NEEDS_LIB_GLIBCONFIG=true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The patch adds new build-time dependencies, but:
- Our CI doesn't respect them (yet), thus if we merge the patch as is, it'll immediately get broken. E.g. what I get if I try to make a build in a docker container as described in the README:
Package atspi-2 was not found in the pkg-config search path.
Perhaps you should add the directory containing 'atspi-2.pc'
to the PKG_CONFIG_PATH environment variable
Package 'atspi-2', required by 'virtual:world', not found
Creating support/modules_libs/jdk.accessibility/libatk-wrapper.so from 16 file(s)
/JetBrainsRuntime/src/jdk.accessibility/linux/native/libatk-wrapper/AtkWrapper.c:25:10: fatal error: glib.h: No such > file or directory
#include <glib.h>
^~~~~~~~
compilation terminated.
(I get the same on Ubuntu 22 before installing the dependencies).
I can suggest 2 following ways of solving the problem:
- to make the corresponding changes to our build scripts, docker images, etc. and include them into this PR ;
- to make the changes provided in this patch disabled by default (i.e. apply
--disable-java-atk-wrapper
by default) and deal with all this CI stuff later after merging the patch .
Anyway, please note @vprovodin noticed earlier that Oracle Linux 8 which we're using as a build platform (via a docker image) doesn't provide these new dependencies, so it may become a serious obstacle for this patch.
- The build instructions in README haven't been updated correspondingly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The build instructions in README haven't been updated correspondingly.
I updated README, but not sure where to locate build instructions
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not sure where to locate build instructions
Our README is currently describing 2 ways of how to build a JBR for Linux:
- Using a specially designed docker container - the chapter Linux (Docker)
- Using an Ubuntu as a host system - the chapter Ubuntu Linux
For now there's probably nothing to change in the Docker chapter, but Ubuntu chapter should be supplemented with the new packages and instructions how to enable/disable ATK wrapper and how to pass custom paths to it. Examples could be Windows's chapters Enable optional NVDA screen reader support and Disable optional JAWS screen reader support.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done at f83061c
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good.
Let's file an issue about fixing CI and enabling back the patch by default and we can close this thread :).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a note: here seems the full list of Ubuntu 22 packages that must be additionally installed to build the patch:
libatspi2.0-dev
libatk1.0-dev
libglib2.0-dev
libatk-bridge2.0-dev
-DATSPI_MINOR_VERSION=$(shell pkg-config --modversion atspi-2 | cut -d. -f2) \ | ||
-DATSPI_MICRO_VERSION=$(shell pkg-config --modversion atspi-2 | cut -d. -f3), \ | ||
LDFLAGS := $(LDFLAGS_JDKLIB), \ | ||
LIBS_linux := -ljvm $(ATK_LIBS) $(AT_SPI2_ATK_LIBS) $(GLIB_LIBS) $(GOBJECT_LIBS), \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider adding -ljava
and/or -lawt
as well if needed (I don't know).
My main question about the entire patch is the following: as you've written in the description, this functionality was removed some time ago due to a lot of bugs mentioned in e.g. IJPL-136231:
and in JBR-4578:
By putting all this logic back, do you state that all those problems have been fixed and now AWT wrapper is in production-ready quality? UPD: still actual crash https://discourse.gnome.org/t/sigsegv-in-g-type-check-instance-is-fundamentally-a-0x11/24285 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I seem to have done reviewing everything related to the Wrapper initialization.
Besides that, this "round" also contains a few comments about the resource management and some general notes.
src/jdk.accessibility/linux/classes/META-INF/services/javax.accessibility.AccessibilityProvider
Outdated
Show resolved
Hide resolved
src/jdk.accessibility/linux/classes/org/GNOME/Accessibility/AtkWrapper.java
Show resolved
Hide resolved
src/jdk.accessibility/linux/classes/org/GNOME/Accessibility/AtkWrapper.java
Outdated
Show resolved
Hide resolved
if (toolkit.getSystemEventQueue() != null) { | ||
toolkit.getSystemEventQueue().push(new EventQueue() { | ||
boolean previousPressConsumed = false; | ||
|
||
public void dispatchEvent(AWTEvent e) { | ||
if (e instanceof KeyEvent) { | ||
if (e.getID() == KeyEvent.KEY_PRESSED) { | ||
boolean isConsumed = AtkWrapper.dispatchKeyEvent(new AtkKeyEvent((KeyEvent)e)); | ||
if (isConsumed) { | ||
previousPressConsumed = true; | ||
return; | ||
} | ||
} else if (e.getID() == KeyEvent.KEY_TYPED) { | ||
if (previousPressConsumed) { | ||
return; | ||
} | ||
} else if (e.getID() == KeyEvent.KEY_RELEASED) { | ||
boolean isConsumed = AtkWrapper.dispatchKeyEvent(new AtkKeyEvent((KeyEvent)e)); | ||
|
||
previousPressConsumed = false; | ||
if (isConsumed) { | ||
return; | ||
} | ||
} | ||
} | ||
|
||
super.dispatchEvent(e); | ||
} | ||
}); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a real need to intercept all the key events? All the keyboard input gets dependent on third-party components' logic we don't control. In the worst cases we may be getting EDT freezes and the inability to use keyboards at all.
Also, I'm not certain how exactly the queues pushing/poping works, but this approach most likely displaces the IntelliJ's own EventQueue installed at https://github.com/JetBrains/intellij-community/blob/794ebfdcb71781022a1357533be9f5fc352408b0/platform/platform-impl/src/com/intellij/ide/IdeEventQueue.kt#L139 .
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is at least one key event that we need to dispatch: IJPL-164705, so we cannot remove it completely.
Yes, Intellij has its own EventQueue and to fix the issue above, we would like to add a new dispatcher using the JBR API: https://code.jetbrains.team/p/ij/repositories/ultimate/revision/a6886190ed6084964ab4b31f01b03b19ccf9c21d. (I will create a new pull request for the JBR API after this one is approved.) It is assumed that it will be execute the code from dispatchEvent(AWTEvent e)
function.
This approach does not replace IntelliJ's queue, quite the opposite, but I think there is no guarantee that it will not. Can we remove the push of a new EventQueue in AtkWrapper() and delegate this responsibility to the swing project?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is at least one key event that we need to dispatch: IJPL-164705
Could you please elaborate which one exactly? It's not clear from the issue description.
Also, if there's only a specific set of keystrokes, let's restrict the logic only to it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At first, I thought the issue was only with pressing the right and left arrow keys, but on Ubuntu 22.04, orca doesn't announce any key events : caret movement, text typing, pressing delete, tab, etc. So, we need to process all key events.
I cannot reproduce this problem on Ubuntu 24.04. If we do not process key events, everything is announced. The difference is in the version of orca.
Also, is it true that we should not consume any events?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See 0fd66b9
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Am I right that the new logic sends all key events both to ATK and to AWT/Swing? Doesn't it mean we can just replace installing of new EventQueue with a new global event listener, just how it's done for window, focus and container events?
Also let me know if some of your questions above still require answers.
BTW, I think each file that gets updated should also have its copyright header updated. One of the most recent examples of our header can be found e.g. in https://github.com/JetBrains/JetBrainsRuntime/blob/f89a6626d56dad71c3429c662bdad89f6b9a0591/src/java.desktop/unix/classes/sun/awt/wl/WLDisplay.java, but I don't know how to properly merge the existing headers with our one. |
…insert` and `text_changed::delete`
1. loadAtkBridge() in static block 2. Changes in initNativeLibrary: - return False instead of exit(1) - throw an exception if initNativeLibrary returns false
1. event doesn't get freed 2. synchronization
/usr/local/include/glib-2.0/gobject/gtype.h:2732: note: this is the location of the previous definition 2732 | #define GTYPE_TO_POINTER(t) ((gpointer) (guintptr) (t)) GOBJECT_AVAILABLE_MACRO_IN_2_80
1. return boolean to throw an exception if something is wrong 2. unref jni_main_loop and jni_main_context if the thread is null 3. undo the effect of jaw_accessibility_init performed a few lines above 4. call initNativeLibrary() and loadAtkBridge() in initAtk()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have finished reviewing the rest of the initial code. Have found a lot of issues, some of them are critical memory leaks or, in opposite, use-after-free errors
else if (javaKeyLocation == KeyEvent.KEY_LOCATION_RIGHT) | ||
javaKeyCode += RIGHT_OFFSET; | ||
|
||
if ((gdkKeyInfo = (GNOMEKeyInfo) keyMap.get(Integer.valueOf(javaKeyCode))) != null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
keyMap
only holds GNOMEKeyInfo
values, so let's make its type more specific and remove the downcast from here and everywhere else (if any).
private static void initializeMap() { | ||
keyMap = new HashMap<>(146); // Currently only 110, so allocate 110 / 0.75 | ||
|
||
keyMap.put(Integer.valueOf(KeyEvent.VK_COLON), new GNOMEKeyInfo(0x20a1, "ColonSign")); // GDK_ColonSign |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[refactoring] There are lots of meaningless integer boxing in this file, let's get rid of them.
public static final class GNOMEKeyInfo { | ||
private int gdkKeyCode; | ||
private String gdkKeyString; | ||
|
||
public GNOMEKeyInfo(int code, String string) { | ||
gdkKeyCode = code; | ||
gdkKeyString = string; | ||
} | ||
|
||
public int getGdkKeyCode() { | ||
return gdkKeyCode; | ||
} | ||
|
||
public String getGdkKeyString() { | ||
return gdkKeyString; | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[refactoring]
public static final class GNOMEKeyInfo { | |
private int gdkKeyCode; | |
private String gdkKeyString; | |
public GNOMEKeyInfo(int code, String string) { | |
gdkKeyCode = code; | |
gdkKeyString = string; | |
} | |
public int getGdkKeyCode() { | |
return gdkKeyCode; | |
} | |
public String getGdkKeyString() { | |
return gdkKeyString; | |
} | |
} | |
record GNOMEKeyInfo(int gdkKeyCode, String gdkKeyString) {} |
// Yes, this is crude, but Java does not provide another way. | ||
String s = e.paramString(); | ||
int begin = s.lastIndexOf("rawCode=") + 8; | ||
int end = s.indexOf(',', begin); | ||
String rawcode_s = s.substring(begin, end); | ||
|
||
keycode = Integer.valueOf(rawcode_s); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This code may end up with exceptions or just give the wrong values e.g. if rawCode
or a comma after it are absent.
I propose to extend sun.awt.AWTAccessor.KeyEventAccessor
with a getRawCode
method and then just:
// Yes, this is crude, but Java does not provide another way. | |
String s = e.paramString(); | |
int begin = s.lastIndexOf("rawCode=") + 8; | |
int end = s.indexOf(',', begin); | |
String rawcode_s = s.substring(begin, end); | |
keycode = Integer.valueOf(rawcode_s); | |
keycode = AWTAccessor.getKeyEventAccessor().getRawCode(e); |
break; | ||
} | ||
default: { | ||
char[] chars = new char[1]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's use String.valueOf(char)
instead
* even been sent! | ||
*/ | ||
static pthread_mutex_t jaw_vdc_dup_mutex = PTHREAD_MUTEX_INITIALIZER; | ||
static jobject jaw_vdc_last_ac = NULL; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are the usages of this variable implemented correctly?
- Comparisons to other JNI references are performed through "raw" pointer comparisons, although different JNI references may reference the same Java object.
- It gets assigned to local JNI reference which are only valid during a JNI downcall.
enum _SignalType { | ||
Sig_Text_Caret_Moved = 0, | ||
Sig_Text_Property_Changed_Insert = 1, | ||
Sig_Text_Property_Changed_Delete = 2, | ||
Sig_Text_Property_Changed_Replace = 3, | ||
Sig_Object_Children_Changed_Add = 4, | ||
Sig_Object_Children_Changed_Remove = 5, | ||
Sig_Object_Active_Descendant_Changed = 6, | ||
Sig_Object_Selection_Changed = 7, | ||
Sig_Object_Visible_Data_Changed = 8, | ||
Sig_Object_Property_Change_Accessible_Actions = 9, | ||
Sig_Object_Property_Change_Accessible_Value = 10, | ||
Sig_Object_Property_Change_Accessible_Description = 11, | ||
Sig_Object_Property_Change_Accessible_Name = 12, | ||
Sig_Object_Property_Change_Accessible_Hypertext_Offset = 13, | ||
Sig_Object_Property_Change_Accessible_Table_Caption = 14, | ||
Sig_Object_Property_Change_Accessible_Table_Summary = 15, | ||
Sig_Object_Property_Change_Accessible_Table_Column_Header = 16, | ||
Sig_Object_Property_Change_Accessible_Table_Column_Description = 17, | ||
Sig_Object_Property_Change_Accessible_Table_Row_Header = 18, | ||
Sig_Object_Property_Change_Accessible_Table_Row_Description = 19, | ||
Sig_Table_Model_Changed = 20, | ||
Sig_Text_Property_Changed = 21 | ||
}; | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This enum is just a copy of AtkSignal.java
. If you mark the constants from there with @Native
annotation, they can be reused directly in native code (a native header org_GNOME_Accessibility_AtkSignal.h
will be generated), thus avoiding the duplications.
Please also check all other native and Java enums for duplicates.
free_callback_para(para); | ||
return; | ||
} | ||
para->child_impl = child_impl; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we increase the reference counter of child_impl
here and then decrease during the cleanup of para
? AFAIU the current code can lead to use-after-free errors.
Please revise all other usages of jaw_impl_get_instance
for ownership sharing errors.
jniEnv, (*jniEnv)->GetObjectArrayElement(jniEnv, args, 1)); | ||
gchar* insert_text = get_string_value( | ||
jniEnv, (*jniEnv)->GetObjectArrayElement(jniEnv, args, 2)); | ||
g_signal_emit_by_name(atk_obj, "text_insert", insert_position, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will g_signal_emit_by_name
free insert_text
? I suspect there are memory leaks here and in the similar case
branches below.
} else if (type == type_released) { | ||
event->type = ATK_KEY_EVENT_RELEASE; | ||
} else { | ||
queue_free_callback_para_event(para); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
event
is leaking here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have reviewed the recent updates.
private static final Object lock = new Object(); | ||
private static AtkWrapperDisposer disposerInstance; | ||
|
||
static { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's convert the class to a lazy initialized singleton, so that threads will be created only when required and not when the class is getting loaded.
AtkWrapper.releaseNativeResources(nativeReference); | ||
obj.clear(); | ||
obj = null; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can these lines be executed out of the synchronized block?
import sun.awt.util.ThreadGroupUtils; | ||
|
||
@SuppressWarnings("removal") | ||
public class AtkWrapperDisposer implements Runnable { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's document the class, like what and how and why it does.
@SuppressWarnings("removal") | ||
public class AtkWrapperDisposer implements Runnable { | ||
private static final ReferenceQueue<Object> queue = new ReferenceQueue<>(); | ||
private static final Hashtable<java.lang.ref.Reference<Object>, DisposerRecord> records = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unused field.
exports sun.java2d to | ||
jdk.accessibility; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this required?
|
||
if (!jaw_log_file) { | ||
perror("Error opening log file " JAW_LOG_FILE2); | ||
return FALSE; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For jbooleans, the proper values JNI_TRUE
and JNI_FALSE
. The same for Java_org_GNOME_Accessibility_AtkWrapper_loadAtkBridge
JAW_DEBUG_I("Thread create failed: %s !", err->message); | ||
g_main_loop_unref (jni_main_loop); | ||
#if ATSPI_CHECK_VERSION(2, 33, 1) | ||
g_main_context_unref(jni_main_context); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure, but shouldn't we do smth like atk_bridge_set_event_context(NULL)
here as well?
if (toolkit.getSystemEventQueue() != null) { | ||
toolkit.getSystemEventQueue().push(new EventQueue() { | ||
boolean previousPressConsumed = false; | ||
|
||
public void dispatchEvent(AWTEvent e) { | ||
if (e instanceof KeyEvent) { | ||
if (e.getID() == KeyEvent.KEY_PRESSED) { | ||
boolean isConsumed = AtkWrapper.dispatchKeyEvent(new AtkKeyEvent((KeyEvent)e)); | ||
if (isConsumed) { | ||
previousPressConsumed = true; | ||
return; | ||
} | ||
} else if (e.getID() == KeyEvent.KEY_TYPED) { | ||
if (previousPressConsumed) { | ||
return; | ||
} | ||
} else if (e.getID() == KeyEvent.KEY_RELEASED) { | ||
boolean isConsumed = AtkWrapper.dispatchKeyEvent(new AtkKeyEvent((KeyEvent)e)); | ||
|
||
previousPressConsumed = false; | ||
if (isConsumed) { | ||
return; | ||
} | ||
} | ||
} | ||
|
||
super.dispatchEvent(e); | ||
} | ||
}); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Am I right that the new logic sends all key events both to ATK and to AWT/Swing? Doesn't it mean we can just replace installing of new EventQueue with a new global event listener, just how it's done for window, focus and container events?
Also let me know if some of your questions above still require answers.
java-atk-wrapper
used to be part of OpenJDK (JDK-8204862, IJPL-136231), but it was removed.java-atk-wrapper
is essential for accessibility in Java applications, including IntelliJ IDEA: IJPL-58438, JBR-4578.java-atk-wrapper
has an LGPL 2.1 license, which can be up-cast to GPL-2.0, allowing the source code to be used.