Jeffrey Emanuel – SmartEdgar, Agents & Markets, Lumera

This project emerged from a simple question I posed to an advanced AI: "How can we use matrix exponentials to improve attention?"

The AI didn't just answer; it proposed a research agenda. It generated 11 exotic mathematical frameworks, scored them on novelty and feasibility, and helped write the JAX kernels to prove them. This is Model-Guided Research: a collaboration where the AI acts as the Principal Investigator.

Here are the most revolutionary architectures we implemented.

1. Matrix Exponential Gauge Learning (Lie Groups)

Standard neural networks operate in vector spaces. But many data manifolds are curved (rotations, scalings).

The Idea: Instead of adding updates ( $x + \Delta x$ ), we multiply by elements of a Lie Group ( $e^A \cdot x$ ).

Generators: We learn the algebra elements (skew-symmetric for rotations, symmetric for scaling).
Exponential Map: $\exp(A) = \sum A^k/k!$ projects these onto the manifold.
Transport: Attention becomes "Parallel Transport" along the sequence.

Why it works: It enforces exact conservation laws (e.g., norm preservation for rotations) and stabilizes gradients in deep networks by preventing exploding/vanishing eigenvalues.

2. Tropical Geometry (Max-Plus Algebra)

What if we replaced multiplication with addition, and addition with max? $a \otimes b = a + b$ $a \oplus b = \max(a, b)$

The Idea: In this "Tropical Semiring," matrix multiplication becomes: $(A \otimes B)_{ij} = \max_k (A_{ik} + B_{kj})$

The Result: A neural network that is piecewise linear by construction.

Robustness: We can mathematically prove margins and robustness certificates.
Efficiency: Attention can be computed without multiplications, potentially unlocking ultra-efficient hardware implementations.

3. Ultrametric Attention (p-adic Numbers)

Standard attention assumes a flat, Euclidean world. But language and code are hierarchical (trees).

The Idea: Use p-adic ultrametric distance: $d(x, y) \le \max(d(x, z), d(z, y))$ In this space, "triangles" are always isosceles. We index tokens into a p-ary tree (trie).

The Mechanism:

Distance = Depth of the Lowest Common Ancestor (LCP).
Attention becomes a query into this tree, routing information only from the relevant branch.
Complexity: $O(N \log N)$ instead of $O(N^2)$ .

4. Braid Group Attention

Sequential data often involves permutations and re-orderings (like code execution).

The Idea: Use Knot Theory.

Tokens are strands in a braid.
Attention interactions are "crossings" ( $\sigma_i$ ).
The network learns a Braid Word that weaves inputs together.

Why it works: Braid invariants provide topological robustness. The network learns features that are invariant to "wiggling" the sequence, capturing the underlying causal structure rather than just position.

5. HOSS: Hyperreal Optimization

We implemented an optimizer based on Nonstandard Analysis (Hyperreal numbers).

It treats gradient steps as infinitesimals ( $\epsilon$ ).
It uses the Transfer Principle to extend first-order logic to infinite precision.
Implementation: A "Langevin-like" update rule that naturally escapes saddle points by probing the loss landscape's curvature at an infinitesimal scale.

The Meta-Result

Beyond the math, this project demonstrated a new way to do science. The AI didn't just code; it invented. It recognized that Tropical Geometry could provide robustness certificates. It saw the link between p-adic numbers and efficient attention.

We are moving from "AI as Tool" to "AI as Co-Author."