NSF Award #2323086

Title: RI: Small: Integrating physics, data, and art-based insights for controllable generative models

Abstract

Generative models refer to a large class of machine learning techniques that can generate user-specified media including images, video, 3D environments, and text — from inputs such as text prompts, sketches, or other user provide images. New generative models are rapidly being developed and are seen as increasingly important in many different applications such as in chatbots and automation. Current generative models are characterized by extremely large models trained on web-scale data, but on closer inspection are found to be unreliable in critically important contexts. This project focuses on generative models for visual media, where current generative models will be advanced by leveraging prior knowledge about how visual features can be described by physical and statistical laws. The sources of knowledge that will be leveraged include physics-based knowledge, insights from traditional content creation techniques, and advances in modeling latent-spaces using novel geometric methods. The anticipated benefits include more robust models, smaller scale models, and more interpretable and modular models.

This research systematically investigating the basics of generative-adversarial networks. The first task considers the role of the input probability distribution from which samples are drawn, generalizing to non-parametric distributions tuned to reduce distribution mismatch under sample mixing. The second task involves architectural novelty in terms of detail layering, where synthesis is broken into a series of simpler architectures. The third task focuses on developing reduced parameter discriminator models, using orthogonality-type constraints as a proxy for physical variables like lighting, texture, and deformation. The fourth task focuses on developing shape-aware architectures, using learnable polynomial basis functions to represent shape more directly. Applications for these methods include augmenting training-sets to create trustworthy machine learning models in contexts such as manufacturing and health, where it is difficult to gather large training sets. Curricular innovations include creating access to these approaches for non-STEM students, in a class titled Machine Learning for Media Arts.

Publications

• Hongjun Choi, Eun Som Jeon, Ankita Shukla, and Pavan Turaga, “Intra-class Patch Swap for Self-Distillation”, accepted at Neurocomputing, 2025.
• Utkarsh Nath, Rajeev Goel, Eun Som Jeon, Changhoon Kim, Kyle Min, Yezhou Yang, Yingzhen Yang, Pavan Turaga, “Deep Geometric Moments Promote Shape Consistency in Text-to-3D Generation”, accepted IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2025.
• Utkarsh Nath, Rajhans Singh, Kuldeep Kulkarni, Ankita Shukla, Pavan Turaga, “Polynomial implicit neural framework to promote geometry awareness in generative models”, accepted International Journal of Computer Vision (IJCV) special issue on Large-Scale Generative Models for Content Creation and Manipulation, 2024.
• Shenyuan Liang, Benjamin Beaudett, Pavan Turaga, Saket Anand, Anuj Srivastava, “Learning Geometry of Pose Image Manifolds in Latent Spaces Using Geometry-Preserving GANs”, accepted at International Conference on Pattern Recognition (ICPR) 2024.
• Utkarsh Nath, Yancheng Wang, Pavan Turaga, Yingzhen Yang, “RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation”, accepted at the International Journal of Computer Vision (IJCV) 2024.
• Eun Som Jeon, Rahul Khurana, Aishani Pathak, Pavan K. Turaga, “Leveraging Topological Guidance for Improved Knowledge Distillation”, accepted at the Geometry-grounded Representation Learning and Generative Modeling (GRaM) workshop on conjunction with the International Conference on Machine Learning (ICML), 2024.
• Baaz Jhaj, Ankita Shukla, Pavan Turaga, Michael Kozicki, “On the impact of pre-training datasets for matching dendritic identifiers using residual nets”, accepted at the International Workshop on Artificial Intelligence for Signal, Image Processing and Multimedia (AI-SIPM) in conjunction with the ACM International Conference on Multimedia Retrieval (ICMR), 2024.
• Kacy Hatfield, Akuadasuo Ezenyilimba, Nitin Verma, Juan Jose Garcia Mesa, So Eun Moon, Elizabeth Tibbetts, Pavan Turaga, Theodore P. Pavlic, “Fine-tuned thin-plate spline motion model for manipulating social information in paper-wasp colonies”, accepted at the CV4Animals: Computer Vision for Animal Behavior Tracking and Modeling in conjunction with Computer Vision and Pattern Recognition (CVPR) 2024.

Public Outreach

Sep 3rd 2025. We recorded an intro lecture for FSE 150 Perspectives on Grand Challenges for Engineering, summarizing recent work on 3D generative AI supported by the grant. The talk is titled “Empowering Artists in the age of 3D GenAI” — a contemporary grand challenge. We focus on ideas and GenAI architectures which promote higher degrees of control to artists. The talk is made publicly available.

Feb 15th 2025. We are proud to partner with Voxel51 to host a VisualAI hackathon at ASU. We had nearly 125 student participants, who also received ASU micro-credentials in three categories: AI Model Development Beginner/Intermediate/Expert.

March 7th 2024. Turaga was invited to a webinar on discussing interdisciplinary insights toward developing guardrails for generative AI and language models. This was hosted by RagaAI, a prime provider of LLM evaluation solutions.