#asttokens — Public Fediverse posts

With support from #ASTTokens, we can then use #radon to collect Raw (LOC, comments, blank lines, etc.) metrics from #Python functions and methods.

After adding support for #wily to annotate source code with these metrics, we can display them inline with help from HTML and JS, as in the attached images.

Each line displays the selected metric (or all metrics) on the left. It allows quickly finding, e.g., code with too few comments or blank lines versus lines of code.

5/5
#codequality

We can now include trailing comments in #Python functions, but want to avoid including extraneous comments in the roundtripped code. To achieve that, we need to stop adding comments as soon as we see one that is less indented than the first line of the function we're interested in.

So we:
1- Recover the code from an #AST node using #ASTTokens.get_text().
2- Add any trailing comments after the last token of the function.
3- Unless the comment is less indented: we stop adding in that case.

4/n

Since #ASTTokens allows getting the next token from end of the function #AST node, it's possible to fetch the next tokens if they are comments, and stop at the last comment before a token of another type.

Using the last comment token's end position, we fetch the #Python source code between what ASTTokens.get_text() gave us and that last comment. That allows a full roundtrip of code that includes trailing comments.

However, that will also include any comments after the function body 😞

3/n

So, for #radon to calculate LOC and number of comments for each #Python function, it needs a way to roundtrip code with all comments included.

Enter #ASTTokens, which "annotates Python abstract syntax trees (ASTs) with the positions of tokens and text in the source code that generated them".

However, trailing comments get lost when using ASTTokens.get_text(), because the comment token isn't linked to the function #AST node. So no roundtrip yet.

2/n

#codequality

https://github.com/gristlabs/asttokens