C-Former: When a Mathematical Theory Derived a Neural Architecture, Predicted Its Failure, and Prescribed the Fix

This paper makes an unusual claim for a machine learning paper: the architecture we present was not discovered through search, not found by intuition, and not adapted from a predecessor. It was derived from a mathematical theory that exists outside the ML canon entirely.

A mathematical framework called Coherence Theory (CT) — originally developed to characterize which patterns persist in systems with finite resources — produced a specific prediction: any sufficiently complex information-processing system should decompose its operations into exactly three orthogonal channels. We took this prediction literally, designed a transformer around it, and tested it.

The architecture failed catastrophically on its first real benchmark (ListOps: 17.4%, near random). But the theory also predicted why it would fail — one of the three channels was mathematically dead — and prescribed the specific correction. The correction produced a 64-percentage-point jump (17% to 81%) from a fix that adds no learnable parameters to the core decomposition.

We present the architecture, the results, and the complete audit trail. The audit trail is not supplementary material. It is the primary evidence for the paper's central claim. A theory that explains results after the fact could be post-hoc rationalization. A theory that guided the actual research — including predicting where things would go wrong and prescribing how to fix them — is doing genuine scientific work.

Transformers route all information through a single representational pathway. Tokens interact via attention, pass through feed-forward layers, and emerge as undifferentiated feature vectors. This architecture is general-purpose but provides no structural decomposition of the signal into interpretable components. The network must discover any useful decomposition from scratch, using parameters and data.

Consider a ListOps expression like [MAX 2 [MIN 4 5] 1]. To solve this, a model must simultaneously handle three structurally different operations: bracket matching (a coordination pattern), operator application (a directed flow), and boundary crossing (the result of an inner expression feeds into an outer one). A standard transformer must learn to separate these three operations from undifferentiated attention patterns. With enough capacity, it does. But what if the architecture already separated them?

Coherence Theory asks a simple question: what does a pattern need in order to persist? CT's answer comes in three parts, each building on the last.

Part 1: The selection inequality. A pattern persists if its benefit exceeds its cost:

For ML practitioners, this is a regularized loss function. CL is task performance (negative loss). The are cost terms. The are regularization weights. Gradient descent on is the selection inequality operating on parameter space.

Part 2: Why exactly three costs. CT proves (from axioms about finite resources, local interactions, and multi-dimensional constraints) that cost decomposes into exactly three orthogonal dimensions. The proof uses a classical result from algebraic topology: the discrete Hodge decomposition theorem.

Part 3: The optimality theorem. CT proves that the total cost function has a unique minimum at independent dimensions. The critical prediction: losing any one of the three channels should produce not a gradual degradation but a catastrophic failure on tasks that require that channel's function.

Each architectural choice traces to a specific CT requirement. This is what makes C-Former different from an architecture found by search: each choice has a reason, and the reason is external to the ML domain.

1. A theory-derived architecture where every structural choice traces to a derivation, not a search. The derivation chain is: CT → three budgets → Hodge decomposition → TD6 tile → C-Former.
2. A theory-predicted failure and theory-prescribed fix. CT diagnosed a dead channel (), predicted catastrophic failure, and prescribed cycle basis injection. The 64pp capability jump (17% to 81%) confirms the prediction.
3. 4/5 LRA wins with 40% fewer parameters across 59 experiments (5 tasks, 3 scales, up to 4 seeds per configuration).
4. Inductive bias as scale substitute. C-Former XS (600K params) beats Standard S (3.77M params) on ListOps — 6x parameter efficiency from structural inductive bias alone.
5. Epoch-1 crystallization and the seed-growth hypothesis. 59.5% accuracy after one epoch (vs. 14.1% standard), with evidence that the Hodge decomposition acts as a representational seed crystal.
6. A complete, published audit trail of assumptions, failures, wrong fixes, and theory-guided corrections as primary evidence.

Let be a connected graph with nodes and edges. Fix an orientation for each edge. The signed incidence matrix has if is the head of edge , if the tail, 0 otherwise. For node potentials : gives the gradient. For edge flows : gives the divergence.

The orthogonal projectors are: and . With designated boundary nodes , the boundary restriction operator extracts flows incident to the boundary, providing a third independent measurement channel.

The balanced Hodge dimensions () ensure neither channel dominates by construction. This is a rare property — most small graphs have unbalanced Hodge dimensions.

Hodge theory in ML. Hodge decomposition for ranking (Jiang et al., 2011), topological signal processing (Barbarossa & Sardellitti, 2020), simplicial neural networks (Bodnar et al., 2021). These apply Hodge decomposition to variable-topology inputs. C-Former uses it as a fixed architectural inductive bias.

Efficient transformers. FNet replaces attention with Fourier transforms (Lee-Thorp et al., 2022). Structured state spaces S4 (Gu et al., 2022) and Mamba (Gu & Dao, 2024) achieve sequence processing. C-Former's multi-tile chain is also but provides interpretable three-channel decomposition.

Geometric deep learning. Group equivariant networks (Cohen & Welling, 2016; Bronstein et al., 2021). C-Former's weight sharing is an instance of group equivariance applied to the tile's internal symmetry.

For a sequence of tokens, create tiles. Each tile processes 6 adjacent tokens on the 6 interior nodes. Edge features combine gradient and cycle components:

where are orthonormal cycle basis vectors (computed once via SVD of , then frozen) and are data-dependent coefficients produced by a small MLP, initialized at scale. Guaranteed: (verified to ).

Each channel receives its own learned projection, ensuring channels process approximately independent information. This satisfies CT's Axiom B4 (independent components have additive costs).

Layer update with momentum ( dynamics, derived from CT's proof that second-order is uniquely optimal). Three-channel normalization replaces LayerNorm: each channel is normalized independently with learned scale ratios, implementing CT's calibration invariance (only cost ratios matter).

For tiles processing tokens: per-layer cost is . Receptive field grows by 6 tokens per layer. Inter-tile boundary exchange creates additional cycles spanning adjacent tiles, enabling long-range coordination.

ListOps at 17.4% is near random for a 10-class task. The architecture derived from first principles was performing worse than chance-level guessing on the task most aligned with its design.

CT analysis identified two provable structural flaws — mathematical identities that guaranteed failure regardless of training.

Flaw 1: Dead cycle channel. Edge features were computed as . By the Hodge theorem, . The cycle projector projects onto . Therefore:

This is not an approximation. It is not a training failure. It is a mathematical identity. The cycle channel carried exactly zero information for all possible inputs. The gradient of the loss with respect to the cycle channel was identically zero.

Flaw 2: B4 violation (shared inputs). All three channels received the same input tensor. CT's Axiom B4 requires independent components to have disjoint supports.

CT's prediction: With only two of three channels active (), the architecture cannot represent cycle-like structure. Restoring the third channel will produce a qualitative capability jump — a phase transition.

Before identifying the root cause as a mathematical identity, we attempted a symmetric cross-product fix: . This is symmetric in the two endpoint features. But cycle flow is inherently antisymmetric. A symmetric product has no orientation and therefore cannot represent cycle flow.

Ablation confirmed: removing the Hodge decomposition entirely improved accuracy by 0.6% with the v1.5 fix. The symmetric product was injecting noise into the cycle subspace.

Three simultaneous corrections, each addressing a specific diagnosed flaw:

1. Cycle injection (fixes dead channel): 12 orthonormal cycle basis vectors from via SVD.
2. Channel independence (fixes B4 violation): Separate learned projections per channel.
3. Multi-tile chain (fixes compression): tiles for sequences of length , with boundary exchange.

The ListOps jump from 17.4% to 81.3% is a 64-percentage-point improvement. The cycle injection itself adds only a ~10K-parameter coefficient MLP (<1% of total parameters). This is not a tuning improvement. It is a phase transition from to — exactly as CT's dimensionality theorem predicted.

All experiments use synthetic data generators matching the LRA task specifications (Tay et al., 2021). Relative comparisons between C-Former and the standard transformer are valid (identical data, hyperparameters, training schedules), but absolute numbers should not be compared directly to published LRA benchmarks. Total compute: approximately $3 on Vast.ai V100 instances. 59 experiments across 5 tasks, 3 scales, and up to 4 seeds per configuration.

Sequential CIFAR-10: 32x32 grayscale pixels flattened to 1024-token sequences.

Why this result is so large. Standard attention is position-agnostic on flattened pixels. C-Former's multi-tile chain preserves spatial locality — each tile processes 6 adjacent pixels, and boundary exchange propagates spatial context. The cycle channel detects local periodic patterns (edges, textures) within each tile.

The standard transformer barely improves with 5x data (it was not data-limited). C-Former's advantage narrows from +3.4pp to +1.45pp. This confirms the advantage is from inductive bias: with sufficient data, the standard model can learn the decomposition that C-Former gets for free from the Hodge structure.

After a single epoch — one pass through 20K training examples — C-Former v3 reaches 59.5% accuracy. The standard transformer is at 14.1%. The broken v1 is at 10.4%.

This is not a normal convergence speedup. A 59.5% epoch-1 accuracy on a 10-class hierarchical parsing task means the three-channel Hodge decomposition, applied to randomly initialized weights, already produces a representation useful for hierarchical parsing before any meaningful gradient has been computed. The frozen Hodge projectors organize random noise into structured signals.

We call this epoch-1 crystallization: the Hodge decomposition acts as a seed crystal, providing an initial organizational structure that gradient descent then refines. C-Former v3 reaches the standard transformer's final accuracy (~78%) at approximately epoch 20.

The crystallization phenomenon aligns with a specific CT prediction about how coherent structures form. CT's seed-growth model says that coherent patterns do not emerge all at once. They begin as small seeds — local regions of high coherence — and grow outward as coherence cascades through connections to neighboring regions. Growth follows the selection inequality: regions where are incorporated; regions where are pruned.

We hypothesize that C-Former's training dynamics literally implement this seed-growth process on the TD6 tile network. The TD6 tile is not merely a processing unit — it is a growth substrate that the learning dynamics use in a way structurally analogous to how CT predicts coherent patterns propagate in physical systems.

The seed-growth pattern bears a structural resemblance to biological morphogenesis. This is an analogy grounded in shared mathematical structure (both systems implement local growth rules under selection pressure), not a claim of biological equivalence.

The value of this analogy is heuristic: it suggests testable predictions. If the seed-growth model is correct, we should observe: (a) dormancy — tiles that stay grey for many epochs then suddenly crystallize; (b) resource competition — in parameter-limited regimes, some tiles “starve”; (c) seasonal variation — growth rate tracks learning rate schedule. These are empirically testable.

UCI HAR dataset (6 classes, 7352 train, 2947 test). The fixed Hodge projectors decompose each input into three budget components.

5 of 6 classes produce physiologically correct decompositions. These profiles are identical across all random seeds because they come from frozen mathematical projectors, not learned weights.

Frozen C-Former backbone (only classifier head trainable: 8.4% of parameters) retains 96.6% of fine-tuned accuracy when transferring from sequence classification to graph property prediction. The three-channel decomposition captures task-general structure that transfers without retraining.

Throughout the research, we tracked the strongest argument against C-Former's value — a practice of deliberate adversarial self-evaluation that CT calls tracking the “anti-binder.”

C-Former demonstrates a methodology alternative to architecture search: derive the architecture from principles about what structures persist, then test the derivation. The value is not that theory-derived architectures are always better. The value is that they are diagnosable. When C-Former failed at 17.4%, CT identified why and what to do. When a hyperparameter-searched architecture fails, the diagnostic is “try different hyperparameters.”

The Hodge decomposition is not specific to C-Former. Any neural architecture that processes information on a graph implicitly mixes three types of flow. Design principle: Architectures that explicitly decompose information flow into orthogonal channels should outperform architectures that mix them, especially at small scale and in noise.

C-Former XS (600K params) beats Standard S (3.77M params) on ListOps — 6x parameter efficiency from structural inductive bias alone. The applications are in resource-constrained settings: edge devices, real-time systems, low-data domains, and privacy-constrained applications.

The dead-channel experience suggests a practical diagnostic: measure whether each channel carries non-zero information. A channel that is mathematically zero is an implementation bug. A channel that converges to zero during training indicates the channel's prior does not match the data's structure. Example: on QM5 molecular data, the cycle channel collapsed to zero because molecular ring aromaticity requires data-dependent cycle structure.

The epoch-1 crystallization (59.5% vs. 14.1%) suggests that initial representation quality matters more than commonly appreciated. If a structured initialization reaches 59.5% before any meaningful gradient update, the optimization landscape it explores is fundamentally different. This connects to lottery tickets (Frankle & Carlin, 2019), neural tangent kernels, and the role of initialization.

Gradient projector: where is the Moore-Penrose pseudoinverse computed via SVD with threshold . Cycle basis: SVD of ; the last 12 columns of (corresponding to zero singular values) form an orthonormal basis for . Both projectors are stored as frozen register_buffer in PyTorch and are never updated by gradient descent.