Expr/SyntaxTree parity: trivial conversions

[mlechu]

As discussed on slack with @c42f and @topolarity, I gave conversions an honest try, but couldn't see the end of the tunnel in terms of correctness. The equivalence between SyntaxNode and Expr is only defined on surface syntax or fully-macro-expanded syntax, and not the steps in between, or if errors occur. We have a plan for syntax evolution now, too, so there compelling reasons not to lump changes to AST layout into JuliaLowering's macro expansion.

Let EST be the "new" Expr-like SyntaxTree I'm working on and CST be the current SyntaxNode-like version. PRs planned:

1. Move SyntaxGraph/SyntaxTree from JuliaLowering to JuliaSyntax #60370
2. (this PR) Trivial EST<->Expr conversion, tested by parsing code in bulk and round-tripping
3. RawGreenNode->EST (identical logic to existing RawGreenNode->Expr, and testable with this PR's Expr->EST)
4. EST->CST*: This would just be a simplification of our existing Expr->CST, and prevents us from needing to make huge changes in desugaring (which understands CST*). Macro expansion will become EST->EST rather than CST->CST*.
5. Before one or both of the above, I may try and merge a basic SyntaxTree pattern-matching macro so operating on these things is less tedious. This is already written, but needs tests.

[topolarity]

Draft due to being atop #60370.

Merged!

EST->CST: This would just be a simplification of our existing Expr->CST, and prevents us from needing to make huge changes in desugaring (which understands CST). Macro expansion will become EST->EST rather than CST->CST*.

At what point do we replace CST with EST completely, if any?

[mlechu]

At what point do we replace CST with EST completely, if any?

PR 4 in the list replaces it completely from the perspective of macro authors. Internal to lowering, we can make individual changes to EST->CST* and desugaring at the same time at whatever pace we'd like.

[topolarity]

Suggested change

	function expr_equal_forgiving(e1, e2; ignore_lnn=true)
	function expr_equal_forgiving(e1, e2; ignore_linenums=true)

in keeping with Base.remove_linenums!

[topolarity]

Maybe a dumb suggestion, but should this do some kind of delta detection to avoid polluting the IR?
(i.e. only a emit a LineNumberNode when the lineinfo changes from stmt-to-stmt)

I'm not sure our debuginfo encoding in Expr is well-defined enough to do that losslessly, but perhaps it's good enough?

[mlechu]

That sounds reasonable, but would be a separate change we could implement here and in both parsers. It isn't so clear to me who is consuming the linenodes and whether they will break.

[topolarity]

Does this handle QuoteNode of non-self-quoting values correctly?

QuoteNode(LineNumberNode(...)), QuoteNode(Expr(...)), QuoteNode(Symbol(...)), etc.

[mlechu]

It should, but I should add separate tests for those.

[topolarity]

Hmm it looks like the LineNumberNode will pollute the debuginfo at least, but also converting the internal value at all does not seem correct to me...

It would require that lowering de-convert it when lowering something like return $(QuoteNode(Expr(...))), but that is also wrong to do, since you don't know whether I actually put a EST (or other SyntaxTree) in there to begin with.

[topolarity]

Very nice test!

[c42f]

At what point do we replace CST with EST completely, if any?

I'd prefer to keep CST and RawGreenNode close together, and aim for CST to eventually be the new AST once we release new-style macros. In the meantime it's an implementation detail which we can keep refining as we see fit.

My very strong preference is that we don't get stuck with the Expr AST heads and semantics in the long term (even if in SyntaxTree form) as it has plenty of deficiencies which I've discussed with @JeffBezanson and tried to fix while developing JuliaSyntax.