Work in progress
The Arithmetic That Ran Before the Program
Rust Conversation R.2 — From Our Syntax Object to Compile-Time Evaluation
Our first macro can preserve this new piece of code as text:
code_string!(add(2, multiply(3, 4)))
It would produce the string-literal expression "add(2, multiply(3, 4))". The macro would still know nothing about the arithmetic.
This time we want an integer evaluator. Its language will be deliberately small:
integer literal
add(expression, expression)
multiply(expression, expression)
Every accepted input must become one i64 literal. eval_integer! does not promise to evaluate an arbitrary Rust expression.
I understand: do not just print strings; compute the actual result.
println("{}", eval_integer!(add(2, multiply(3, 4))));
14
Based on your description, the eval_integer macro needs to expand into code containing the literal 14.
Can it expand into
2 + (3 * 4)
?
I would say no. Although they are equal, that is not the same code as 14.
Yes. In the macro logic, you are not just creating code; you must also complete the arithmetic. To do that, we need some way to convert the Rust code into actual Rust values. We begin by designing a type that can hold these expressions. In compiler theory, this is usually called an abstract syntax tree.
I would use:
enum IntegerExpr {
Literal { value: i64 },
Call(IntegerCall),
}
struct IntegerCall {
function: String,
arguments: Vec<IntegerExpr>,
}
Very close, but there is an important distinction between an AST type and an ordinary data type. An AST type must be defined either as an atomic AST type, such as the number 2, or as a struct or enum whose fields are themselves AST types.
Suppose I want to store the code 2. What is its atomic AST type?
It is LitInt. You have already seen the AST type for a string: LitStr. Revise your definition.
enum IntegerExpr {
Literal { value: LitInt },
Call(IntegerCall),
}
struct IntegerCall {
function: LitStr,
arguments: Vec<IntegerExpr>,
}
How can we convert written code into this AST?
Ask Syn to produce the type named on the left:
let expression: IntegerExpr = syn::parse2(input)?;
It does not compile:
the trait `Parse` is not implemented for `IntegerExpr`
The Rust type exists, but Syn does not yet know how to map the input into it. Do we have to write a parser ourselves?
Not for these regular shapes. We use syn_derive, a helper crate whose procedural macros generate the mappings from our type definitions. Our project usually requests both directions:
#[derive(syn_derive::Parse, syn_derive::ToTokens)]
Parse maps tokens into our value. ToTokens lets a later compiler stage put that value back into code.
So while we build eval_integer!, another procedural macro generates part of its implementation. A macro inside our macro project!
Exactly. syn_derive::Parse runs when this macro crate is compiled and generates ordinary Rust code. Later, eval_integer! calls that generated code when it receives tokens from its caller.
For a struct, the generated mapping can read its fields in order. But the input contains neither the word Literal nor the word Call. How can it choose an enum variant?
It cannot infer that choice. We mark the first alternative with a rule:
enum IntegerExpr {
#[parse(peek = LitInt)]
Literal { value: LitInt },
Call(IntegerCall),
}
#[parse(peek = LitInt)] asks whether the next token has the LitInt shape without consuming it. If it does, select Literal. If it does not, continue to Call.
Then I can simulate it:
input 2 -> peek sees LitInt -> choose Literal -> value consumes 2
input add(...) -> next token is not LitInt -> continue to Call
So peek only chooses the alternative. The selected variant's fields still consume and store the input.
examples/rust-02-compile-time/compile-time-macros/src/integer.rs
Open lines 7–14
#[derive(syn_derive::Parse, syn_derive::ToTokens)]enum IntegerExpr { #[parse(peek = LitInt)] Literal { value: LitInt, }, Call(IntegerCall),}
Exactly. Call has no peek annotation because it is the final fallback. Its fields must still match the remaining input. For example, input beginning with + reaches Call but then fails because its first field expects an identifier.
This one-token distinction is enough because we deliberately designed the two forms to begin differently. If two alternatives began the same way, we would need a different or more detailed selection rule. That general theory is outside this tutorial.
The Call(IntegerCall) variant delegates the remaining tokens to IntegerCall. My current version begins with function: LitStr, but the input contains add, not "add". Are those the same syntax shape?
*Optional theory.* The general study of choosing a top-down grammar alternative with bounded lookahead is called LL(k) parsing. See Rosenkrantz and Stearns, *Properties of Deterministic Top-Down Grammars*90446-8) (1970).
No. LitStr represents a quoted Rust string literal. A bare name such as add is an identifier, represented by Syn's Ident:
function: Ident,
After that field consumes add, the remaining input is (2, 3). Does Vec<IntegerExpr> say where that list begins or which token separates its elements?
No. Vec can store the expressions after they have been found, but it does not describe the written parentheses or commas.
Then evolve that field into a syntax-aware, comma-separated list:
#[syn(parenthesized)]
paren_token: token::Paren,
#[syn(in = paren_token)]
#[parse(Punctuated::parse_terminated)]
arguments: Punctuated<IntegerExpr, Token![,]>,
parenthesized recognize (...) and keep its delimiter
in = paren_token read the next field from inside (...)
Punctuated<IntegerExpr, Token![,]> keep expressions separated by commas
parse_terminated use Syn's existing list mapping
Unlike Vec, Punctuated retains the separators so ToTokens can write the same call shape back into code.
Let me trace add(2, 3) from left to right:
add -> function: Ident
(2, 3) -> paren_token opens the nested input
2, 3 -> arguments: Punctuated<IntegerExpr, Token![,]>
Each argument is mapped as another IntegerExpr. Now every written part has a destination.
examples/rust-02-compile-time/compile-time-macros/src/integer.rs
Open lines 18–26
#[derive(syn_derive::Parse, syn_derive::ToTokens)]struct IntegerCall { function: Ident, #[syn(parenthesized)] paren_token: token::Paren, #[syn(in = paren_token)] #[parse(Punctuated::parse_terminated)] arguments: Punctuated<IntegerExpr, Token![,]>,}
Both IntegerExpr and IntegerCall now derive the two mappings. Try the line that failed before:
let expression: IntegerExpr = syn::parse2(input)?;
Now it works. parse2 invokes the generated Parse implementation, which selects a variant and fills each syntax-aware field. Tokens have become our IntegerExpr.
Now the input is an IntegerExpr. How would you define its value from the two enum alternatives?
A literal produces the integer it represents. A call first evaluates its arguments, then applies the named operation:
Literal(n) -> n
Call add(left, right) -> evaluate(left) + evaluate(right)
Call multiply(left, right) -> evaluate(left) * evaluate(right)
The call cases are recursive because their arguments are themselves IntegerExpr values.
Write only that enum dispatch first.
value.base10_parse() is new. Does it turn the stored LitInt syntax into the ordinary integer value needed by evaluation, while the Call case delegates to a helper?
examples/rust-02-compile-time/compile-time-macros/src/integer.rs
Open lines 63–68
fn evaluate(expression: &IntegerExpr) -> syn::Result<i64> { match expression { IntegerExpr::Literal { value } => value.base10_parse(), IntegerExpr::Call(call) => evaluate_call(call), }}
Yes. base10_parse() is Syn's conversion from an integer-literal syntax value to a Rust integer. For calls, the helper renders the stored Ident as text, pairs it with arguments.len(), and matches the supported name-and-arity combination. Which two combinations should have meaning?
("add", 2) should evaluate its two children and add them. ("multiply", 2) should evaluate its two children and multiply them. Any other name or argument count should be rejected.
So the derived mapping accepts the general shape of a call, while our evaluator decides which call names and arities have meaning.
examples/rust-02-compile-time/compile-time-macros/src/integer.rs
Open lines 74–75
("add", 2) => Ok(evaluate(&call.arguments[0])? + evaluate(&call.arguments[1])?), ("multiply", 2) => Ok(evaluate(&call.arguments[0])? * evaluate(&call.arguments[1])?),
We now have the token-to-IntegerExpr mapping, the evaluator, and Quote from R.1. How should expand_expression connect them?
let expression: IntegerExpr = syn::parse2(input)?;
let value = evaluate(&expression)?;
Ok(quote! { #value })
Tokens become our IntegerExpr, ordinary Rust evaluates it to an i64, and Quote turns that value back into code.
examples/rust-02-compile-time/compile-time-macros/src/integer.rs
Open lines 39–43
pub(crate) fn expand_expression(input: TokenStream) -> syn::Result<TokenStream> { let expression: IntegerExpr = syn::parse2(input)?; let value = evaluate(&expression)?; Ok(quote! { #value })}
With the notes folder open in VS Code, expand eval_integer!(add(2, multiply(3, 4))) using rust-analyzer: Expand macro recursively at caret.
The expansion is 14i64. The call-shaped arithmetic has disappeared; only the computed literal remains.
Now run the caller:
cargo run --manifest-path examples/rust-02-compile-time/Cargo.toml --package compile-time-demo
14
The executable prints the value of the generated literal. The expansion—not only the printed result—is the evidence that the macro performed the arithmetic.