I develop frameworks for understanding how language models build representational capacity across layers, token regimes, optimizers, and model scale. While classical scaling laws relate loss to compute, my work characterizes how internal capacity itself scales through nonlinear eigenspectrum dynamics, spectral scaling laws, and optimizer-induced capacity allocation. A key finding is that optimizer choice, not only architecture, determines how much nominal capacity a model actually realizes: models with nearly identical validation loss can still differ sharply in their internal representation geometry. This points toward architecture-optimizer co-design, where architectural choices define available representational degrees of freedom, while optimizer geometry governs which directions become active, variance-carrying, and ultimately realized during training.

I design neural architectures, training methods, and regularization techniques for efficient private inference, where models perform inference directly on encrypted data. This work targets the latency and communication overhead introduced by nonlinear operations, and develops inference-efficient substitutes for these components without sacrificing model quality. In language models, removing nonlinearities can distort attention dynamics, producing entropy collapse in deeper layers, which destabilizes training, and entropic overload in earlier layers, which leaves attention heads under-utilized. I address these failures through static normalization alternatives and hierarchical entropy regularization, which restore attention stability and preserve head diversity under private-inference constraints. This line of work uses encrypted inference as a lens to understand how nonlinearities regulate information flow and stabilize attention dynamics.

My earlier work explored hardware-aware co-design for DNNs through roofline performance modeling and data-reuse-aware compact architectures. I found that conventional arithmetic-intensity metrics can obscure the data-movement structure that determines inference efficiency, since weights and activations have different reuse patterns across architectures and accelerator memory hierarchies. To capture these effects more faithfully, I proposed data-reuse-aware arithmetic intensity. This systems background guides how I think about compute bottlenecks, data movement, and the interaction between algorithms, model structure, and hardware.