Use executing and asttokens

[alexmojaki]

This brings in two dependencies to do the heavy lifting for debug:

• https://github.com/alexmojaki/executing finds the AST node
• https://github.com/gristlabs/asttokens converts AST nodes to source code

Both are very well tested and reliable and are used in several other libraries with no issues, including https://github.com/gruns/icecream. You'll note that several test cases no longer give warnings. In particular this fixes #47 and #71.

executing can identify the AST node in almost all situations, which means:

• return the arguments to debug() #69 will be easy to implement, not just in variable assignments but also in things like list comprehensions as a commenter mentions.
• Any function name is allowed, e.g. from devtools import debug as dbg will work. In fact the function can be any expression.

[samuelcolvin]

Thanks a lot, I'll look soon. What is the performance impact?

[alexmojaki]

executing caches the process of identifying the node, so repeated calls are very fast. This code:

from devtools import debug

with debug.timer():
    for i in range(10000):
        debug.format(i + i)

is about 5 times faster now.

[alexmojaki]

The tests are failing because Python 3.8 changed the location for generator expressions, so in older versions parentheses are not included: gristlabs/asttokens#50

Shall I differentiate tests based on version? Or just remove the generator expression from the test?

[alexmojaki]

In case you were put off by the generator issue, I'll clarify. The problem is that in a script like this:

from devtools import debug

debug(x for x in [1, 2])

it's not clear what should be considered the source code of the generator expression. In Python 3.7 you get:

script.py:3 <module>
    x for x in [1, 2]: (
        1,
        2,
    ) (generator)

In 3.8:

script.py:3 <module>
    (x for x in [1, 2]): (
        1,
        2,
    ) (generator)

The only difference is the surrounding parentheses. As far as devtools is concerned, this feels like a pretty unimportant philosophical question. On top of that, putting a generator inside debug() is already a questionable thing to do. So I think it'd be fine to remove this case from the tests.

[samuelcolvin]

Sorry I haven't had time to review this.

On the generators, the behaviour seems absolutely fine. But I'd prefer to keep tests, just have a test for each case and skip them in the other case.

[samuelcolvin]

This looks amazing. I'm so sorry it's taken me so long to get around to reviewing it.

I've had a new job and a new baby, but I know it can still be very annoying when thoughtful and time consuming work like this get's ignored for so long. In short, mea culpa 🙏.

I'm going to make a few tiny changes, merge it and make a new release.

[samuelcolvin]

okay, thanks. I'm trying to see if I can easily remove that function, if not I'll revert.

[samuelcolvin]

hi @alexmojaki,I have a related question for you since you seem to know the python metaprogramming space pretty well - do you know of any faster alternative for highlighing python than pygments?

Much of devtool's poor performance is due to pygments (often >90%). I therefore thinking of building package to wrap https://github.com/trishume/syntect, but I don't want to if something similar already exists.

[alexmojaki]

No I don't know about any alternatives to pygments

[samuelcolvin]

thanks so much for this.

I'll make a new release today.