-
Notifications
You must be signed in to change notification settings - Fork 567
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial support for QtWebEngine and Async callbacks for timeline/Qt Integration #3604
Conversation
…ntegration. - Requires QtWebEngine and Qt 5.6+ - Requires OpenGL python module (to fix black screens on some systems) - Enables smooth scrolling on timeline
This is going to fail on almost all build servers for a little while. My current plan is...
Any suggestions or patches are welcome! 🙏 Thx! |
@jonoomph But, due to whatever other changes you've made to the freeze code in the meantime, it doesn't look like this branch's |
@ferdnyc Yeah, I ended up using the same version ( |
…webengine-support
Quick Update: I have a working AppImage created on our Bionic build server! I've tested the AppImage with Ubuntu 18.04 and 20.04. However, I'm pretty sure that further tweaks will be needed for wider compatibility. @ferdnyc Perhaps you can test the AppImage (from GitLab build artifacts on the On a side note, we might consider switching our build process for Linux to use https://github.com/probonopd/linuxdeployqt, in the future. It looks pretty nice, and will simply our build-server.py script quite a bit, and will probably yield a much more intelligent AppImage, with respect for Qt required files, and required dependencies. |
cmd = JS_SCOPE_SELECTOR + ".setAudioData('{}',{});".format( | ||
right_clip.id, self.waveform_cache.get(right_clip.id)) | ||
self.page().mainFrame().evaluateJavaScript(cmd) | ||
self.run_js(JS_SCOPE_SELECTOR + ".setAudioData('{}',{});".format(right_clip.id, self.waveform_cache.get(right_clip.id))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Aw. I was really hoping to keep those lines narrow enough that they didn't require horizontal scrolling just to read them on GitHub. (Which was the whole point of splitting them up — not the GitHub part, just the length in general.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whoops, sorry. I was trying to simplify things as much as possible, but I also don't want crazy long lines.
# Not ready, try again in a few milliseconds | ||
if retries > 1: | ||
log.warning("TimelineWebView::eval_js() called before document ready event. Script queued: %s" % code) | ||
QTimer.singleShot(100, partial(self.eval_js, code, retries + 1)) | ||
# Not ready, try again in a few moments | ||
log.error("TimelineWebView::run_js() called before document ready event. Script queued: %s" % code) | ||
QTimer.singleShot(50, partial(self.run_js, code, callback)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this less likely to happen, with QtWebEngine? (He asks, hopefully.) Because, on my system, OpenShot used to log that message a minimum of four times — sometimes six to eight times — at each launch, when it was using the old values and didn't skip retries. Which is incredibly annoying and unnecessary.
I'm also not crazy about logging an error for something that we expect to happen — to the point that we just retry the operation, proceeding as if everything's fine. Is there something wrong, or is it expected? I feel like logging it as an error sends mixed messages — heck, even a warning may be excessive.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might set a threshold for the number of tries we expect might fail, and only log warnings after X # of attempts. That would still show an error in extreme cases where the document.ready event fails for some reason. I like that approach. 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might set a threshold for the number of tries we expect might fail, and only log warnings after X # of attempts. That would still show an error in extreme cases where the document.ready event fails for some reason. I like that approach.
...So, exactly what the previous code used to do? #JustSayin 😉
# Init javascript bounding box (for snapping support) | ||
code = JS_SCOPE_SELECTOR + ".startManualMove('{}', '{}');".format(self.item_type, self.item_id) | ||
self.eval_js(code) | ||
self.run_js(JS_SCOPE_SELECTOR + ".getJavaScriptPosition({}, {});".format(event_position.x(), event_position.y()), partial(callback, self, data)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as earlier comment, re: the width.
try: | ||
# Solution to solve QtWebEngineWidgets black screen caused by OpenGL not loaded | ||
from OpenGL import GL | ||
except ImportError: | ||
pass |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm. I don't know about this. There isn't even a Python module OpenGL
on my system.
I wonder if the issue is the QtOpenGL setup not being done at application start? There was something in the documentation about that, I remember having trouble with that during one of my abortive experiments. (Specifically, the afternoon I spent unsuccessfully trying to get openshot-player
to render to a QOpenGLWidget
.)
Apparently, right after creating the QApplication
, you should configure the application's default OpenGL surface handling. In C++ that was:
QApplication a(argc, argv);
// Configure all OpenGL contexts
QSurfaceFormat glFormat;
glFormat.setSwapBehavior(QSurfaceFormat::DoubleBuffer);
QSurfaceFormat::setDefaultFormat(glFormat);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, I'll give this a try! I figured the PyOpenGL was just a work-around until I figured out what was happening. Without it, I only get a black rectangle for the timeline.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mmm. Well, if it works, maybe it doesn't hurt anything to have it there regardless.
My system now does have an OpenGL
module, because PyOpenGL
was added to the requirements.txt
when pipreqs
scanned the source tree, so it's not like it was a really difficult dependency to satisfy.
(Of course, it's only present in the source to be found because of this import
line, so the whole thing's sort of circular.)
src/timeline/app.js
Outdated
// Bind to keydown event (to detect SHIFT) | ||
body_object.keydown(function (event) { | ||
if (event.which === 16) { | ||
if (timeline) { | ||
timeline.qt_log("Shift pressed!"); | ||
} | ||
body_object.scope().shift_pressed = true; | ||
} | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, funny story about these. Turns out, scope()
is a debugging tool, and we'd actually stand to make a significant performance gain if we could turn off Angular debugging on the Timeline. But then scope()
wouldn't work from JQuery.
Unfortunately, I have tried desperately to get shift-tracking working with AngularJS's own ng-keydown
and ng-keyup
, and I have never gotten it to work anything close to reliably.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, if we could disable debugging, but still somehow access Angular, that would work fine. But I have no idea how to do that. If I have time, I'll do some research on this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking, in theory we could add hidden <input>
s to the HTML, and store the data involved in those. Then, JQuery could update the values without needing scope()
, and Angular could access them directly... well, however we want, really. Using jQuery selectors, using angular.element()
, etc.
I'm thinking something like,
<html><body>
<stuff stuff stuff>
<input type="hidden" id="shift_state" value="0">
</stuff stuff stuff>
<script>
$("body").keydown(function (event) {
if (event.which === 16) $("#shift_state").val(1);
});
$("body").keyup(function (event) {
if (event.which === 16) $("#shift_state").val(0);
});
</script>
<script>
// currently, the only place where shift_pressed is ever read...
function moveBoundingBox(scope, previous_x, previous_y, x_offset, y_offset, left, top) {
var snapping_result = Object();
snapping_result.left = left;
snapping_result.top = top;
// Check for shift key
if (parseInt($("#shift_state").val(), 2) === 1) {
// freeze X movement
x_offset = 0;
}
</script>
I mean, I can tell you that with the timeline loaded in a browser window and the debugging console open, I can see the value=""
attribute of the <input>
flipping from "0"
to "1"
to "0"
as I press and release Shift, so it at least works to that extent...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, if we could disable debugging, but still somehow access Angular, that would work fine. But I have no idea how to do that. If I have time, I'll do some research on this.
I was thinking, in theory we could add hidden
<input>
s to the HTML, and store the data involved in those. Then, JQuery could update the values without needingscope()
, and Angular could access them directly... well, however we want, really. Using jQuery selectors, usingangular.element()
, etc.
...Actually I suspect there's no point in doing that, and I doubt we'll be able to get away from scope()
any time soon. I didn't realize until yesterday that the entire Qt < == > JS interface depends on scope()
:
openshot-qt/src/windows/views/timeline_webview.py
Lines 55 to 56 in 0058f87
# Constants used by this file | |
JS_SCOPE_SELECTOR = "$('body').scope()" |
Maybe if we were to rewrite their entire communication layer as a WebChannel, or something like that, then we could get away from scope()
. But for now, it seems it's here to stay.
# Callback function, to redraw audio data after an update | ||
def callback(self, clip_id, callback_data): | ||
has_audio_data = callback_data | ||
print('has_audio_data: %s' % has_audio_data) | ||
|
||
if has_audio_data: | ||
# Re-generate waveform since volume curve has changed | ||
self.Show_Waveform_Triggered(clip_id) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not put this right before the run_js()
call it's passed to? There's no reason it needs to be defined right at the top of the function; seems like it just makes the code harder to read.
Ooh, hey — if self
is only used to look up the function the callback is going to call, could you just pass that directly, instead? Then the callback can be just a simple function, instead of looking like a method. I'm thinking something like:
# ...prior stuff in function...
def cb(clip_id, fn, result):
if result:
fn(clip_id)
self.run_js(JS_SCOPE_SELECTOR + ".hasAudioData('{}');".format(clip.id),
partial(cb, clip.id, self.Show_Waveform_Triggered))
I have to test that, but it should work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hrm, wait, that may not work — now I'm thinking that it loses the self
argument to Show_Waveform_Triggered
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, hey, it does work! Python is smart enough to know that self.function_name
, when passed as a function object, effectively represents a call to function_name(self, ...)
— the value of self
doesn't need to be explicitly carried alongside it:
Python 3.8.3 (default, May 29 2020, 00:00:00)
[GCC 10.1.1 20200507 (Red Hat 10.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import functools
>>>
>>> class Thinger:
... def thing(self, count=0):
... print(["thing! " for x in range(0,count)])
... def indirect(self, do_thingy=False):
... def cb(fn, do_it, val):
... if (do_it):
... fn(val)
... return functools.partial(cb, self.thing, do_thingy)
...
>>> a = Thinger()
>>> doit = a.indirect(True)
>>> doit(2)
['thing! ', 'thing! ']
>>> doit(5)
['thing! ', 'thing! ', 'thing! ', 'thing! ', 'thing! ']
I think we should definitely move away from the old, long-since-disavowed With cx_Freeze 5.1.1, though, this does build with Bionic — not with Xenial, since there's no QtWebEngine. So I added the Travis Xenial build to |
Oh! Actually, this built with cx_Freeze 6.1 — it was the Xenial build that failed, but not because of cx_Freeze. I didn't even think to check the build logs. So, going back up to 6.1 should be fine, now that Xenial is |
Downloadable Windows Qt (The Qt installer version) can include WebEngine, right? If so, switching to the binary PyQt wheels may work, instead of using the MSYS2 packages. As long as libopenshot is compiled against the same Qt version, it should be compatible with a different packaged binary Qt. |
Actually, what am I saying? You don't even need to use the Qt installer. If I get my libopenshot packaging sorted out, that could really make this simple, but even without that it should hopefully just be a matter of:
|
I have a Windows 10 VM now, and 20GB of system RAM so I can actually run it without having to shut down everything else on my system. I'll play around with it a bit. (Edit: Of course, now I'm reminded that I haven't even instaled MSYS2 on this VM yet, never mind any of the other parts. This may take a bit longer than I thought.) |
Hmmmmm. In fact, now I'm seeing this |
@ferdnyc I don't believe that package contains QtWebEngine (and related files). It looks like they removed all the important web engine bits. |
Doesn't need to! libopenshot has no need for QtWebEngine. Only |
CRIPES, |
@ferdnyc FYI: I sent you an email with the Windows builder info. 👍 |
@jonoomph Yeah, saw that go by, haven't looked yet. Sorry. Figured while I'm experimenting, at least, it made more sense to work locally where I could do insane things like install all 10gb of Visual Studio just to poke around in its bundled Python. So far, it's... not going as well as I'd like. I haven't given up yet, but I also haven't succeeded yet and I'm not even certain I can. My plan may yet prove unworkable. The sticking point is Python, which is very picky about how its extensions are compiled. Windows 10, MSYS, and MinGW are three completely different platforms to Python, and compiled extensions built for one of them can't run in any of the others. (TBC, posting this much from my phone so I can continue on my desktop...) |
Right, so...
Binary wheels of PyQt5 exist only for Windows-native Python. Not for MinGW Python, not for MSYS Python. And if So, that's a non-starter, and in terms of pre-built PyQtWebEngine, Windows Python is the only game in town. And that's... not great, because it means we need a I think we could get away with the DLLs, probably (though I'm not 100% sure on that either). But to use libopenshot with Windows Python at least the SWIG bindings would have to target that Python directly. So, right now I'm... exploring a few different possible approaches to that.
That last one is probably a certain path to success in terms of the goal, but I'm not sure how difficult the effort would prove, or how successful we're likely to be. But, advantage: We'd get to use the binary distributions of both Qt and PyQt5, without having to build anything ourselves. Still, I'm not sure I really want to go down either of those last two roads, should the earlier options all fail. (Part 3 to follow...) |
There is, however, one light in the tunnel that's not an oncoming train. As I was reading up on all this 'round the web, time and time again I kept running into the same frustrated comments from people in the same boat we're in: Because of the lack of workable free-software-stack QtWebEngine support on Windows, they still can't get rid of their old QtWebKit implementation, and they're forced to maintain dual support for both backends simultaneously. Well, I'm here to tell you that it's impossible to read that more than like 10, maybe 15 times without eventually noticing that it ALSO applies to you as well. None of the QtWebEngine changes in the OpenShot So, the worst-case scenario for us, really, is that we end up having to support both backends, and really as worst-cases go that's not so bad, as it's pretty doable. So, regardless what happens with QtWebEngine on Windows, just because it's a good idea ANYWAY I think it makes a lot of sense to take a page from the PIMPL rulebook, and uncouple That'd also allow us to make other improvements. Like, we could build our cmd = JS_SCOPE_SELECTOR + ".setScale(" + str(newScale) + "," + str(cursor_x) + ");"
self.eval_js(cmd) we'd just do: run_js('setScale', str(newScale), str(cursor_x)) And trust it to assemble the JS call appropriately. It'd know to quote all the strings and run_js('someCall', 'anArg', { "another": "arg", "itsa": "dict" }, callback=self.whatever) and that gets turned into: self.page().runJavaScript(
'$(\'body\').scope().someCall("anArg", { "another": "arg", "itsa": "dict" });',
callback) automatically. Why wouldn't we want that? Regardless! It's like with climate change, the worst-case scenario for responding to it is, "What if we clean up the environment and make the world a better place to live for no reason?" Yeah, oh no. That'd be just awful. 😑 So, I'm excited enough about this idea, and I think it makes enough sense to do even if we DON'T need to for QtWebEngine, that I'll probably start working on that as soon as I get frustrated enough with the building-libopenshot-for-Windows-Python thing. |
@ferdnyc YES, I'm a big fan of what you are describing. 🤯 It seems like it would be a fairly minor change, to abstract the classes a bit, and make it seamlessly fallback to Webkit if WebEngine fails to import. This would buy us some time, and maybe |
@ferdnyc Are you going to give this approach a try? Or should I begin work on it? Just making sure we don't step on each other. Thx! |
@jonoomph I've started poking at it, I already have the But I'll push early and often, once I've got things at least semi-working, so you can check it out and hopefully spot whatever I end up missing. |
Conditional inheritance? ehh?
|
Or perhaps have 2 possible Mixin classes?
|
I actually don't think there will need to be any big split on the JS side, it seems like the JS code could be all but identical to support either/both. (Save for a very thin wrapper around the whole thing, conditionally loading The real divide is going to be in the Qt classes. I've spent the past 25 minutes trying to talk myself out of admitting the truth: that we should really have both a Like I said, I've been trying to come up with reasons not to do that, but it's hard to do because they're all wrong and it's the right approach. So... I'm sure I'll see reason eventually. |
No reason we can't extract out the Inheritance and unique initialization from our current Timeline class, and then come back later with a more complex refactor (model / view). In other words, we can get it fixed now and building, which unblocks all the webengine PRs and keeps our daily builds working for now. Then we can go all out and work on the model / view implementation. How does that sound? |
…ill be found and will work)
Creating a mixin class to support both WebKit and WebEngine
Based on the initial work: #2512
Related to PRs:
OpenShot/libopenshot-audio#100
OpenShot/libopenshot#549