I develop frameworks for understanding how language models transform and allocate representational capacity across layers, token regimes, optimizers, and model scale. While classical scaling laws relate loss to compute, my work studies how internal capacity itself scales through nonlinear eigenspectrum dynamics, spectral scaling laws, and optimizer-induced spectral capacity. A key finding is that the optimizer, not only the architecture, determines how much nominal capacity a model actually realizes: models with nearly identical validation loss can still differ sharply in their internal representation geometry. This points toward architecture–optimizer co-design, where architectural choices define the available representational degrees of freedom, while optimizer geometry shapes which directions become active, variance-carrying, and ultimately realized during training.

I design neural architectures, training methods, and regularization techniques for efficient private inference — inference performed directly on encrypted data. This work studies the latency and communication overhead incurred by nonlinear operations, and develops inference-efficient substitutes for these components without sacrificing model quality. In language models, I show that removing nonlinearities can destabilize attention dynamics, producing entropy collapse in deeper layers and entropic overload in earlier layers. I address these failures through static normalization alternatives and hierarchical entropy regularization, which restore attention stability and preserve head diversity under private-inference constraints. Taken together, this work uses private inference as a lens to understand how nonlinearities regulate information flow and stabilize attention dynamics.

My earlier work studied hardware-aware co-design for DNNs through roofline performance modeling and data-reuse-aware compact architectures. I showed that conventional arithmetic-intensity metrics can obscure the data-movement structure that determines inference efficiency, since weights and activations have different reuse patterns across architectures and accelerator memory hierarchies. To capture these effects more faithfully, I proposed data-reuse-aware arithmetic intensity. This systems background shapes how I think about compute bottlenecks, data movement, and the interaction between algorithms, model structure, and hardware.