Workshop
Benchmarking Machine Learning Workloads on Emerging Hardware
Tom St John · Murali Emani · Wenqian Dong
Thu 8 Jun, 5 a.m. PDT
With evolving system architectures, hardware and software stack, diverse machine learning workloads,
and data, it is important to understand how these components interact with each other. Well-defined
benchmarking procedures help evaluate and reason the performance gains with ML workload to system
mapping.
Key problems that we seek to address are: (i) which representative ML benchmarks cater to workloads
seen in industry, national labs, and interdisciplinary sciences; (ii) how to characterize the ML workloads
based on their interaction with hardware; (iii) what novel aspects of hardware, such as heterogeneity in
compute, memory, and bandwidth, will drive their adoption; (iv) performance modeling and projections
to next-generation hardware.
The workshop will invite experts in these research areas to present recent work and potential directions
to pursue. Accepted papers from a rigorous evaluation process will present state-of-the-art research
efforts. A panel discussion will foster an interactive platform for discussion between speakers and the
audience.
Schedule
| 
            
                Thu 5:00 a.m. - 5:10 a.m.
            
         | Introduction
        
        
        
            
                (
                
                    Opening Remarks
                    
                
                )
            > | 🔗 | 
| 
            
                Thu 5:10 a.m. - 5:55 a.m.
            
         | Towards More Efficient Vision Transformers: From Novel Few-Shot Parameter-Efficient Tuning to New Linear-Angular Attention - Yingyan Lin (Georgia Tech)
        
        
        
            
                (
                
                    Invited Talk
                    
                
                )
            > SlidesLive Video | 🔗 | 
| 
            
                Thu 5:55 a.m. - 6:20 a.m.
            
         | Understanding Time Variations of DNN Inference in Autonomous Driving - Liangkai Liu (Wayne State University)
        
        
        
            
                (
                
                    Presentation
                    
                
                )
            > SlidesLive Video | 🔗 | 
| 
            
                Thu 6:20 a.m. - 6:45 a.m.
            
         | Chakra: Advancing Performance Benchmarking and Co-Design Using Standardized Execution Traces - Srinivas Sridharan (Meta)
        
        
        
            
                (
                
                    Presentation
                    
                
                )
            > SlidesLive Video | 🔗 | 
| 
            
                Thu 6:45 a.m. - 7:10 a.m.
            
         | Performance Analysis of Binary Neural Networks Deployed in NVM Crossbar Architectures - Ruirong Huang (Cornell)
        
        
        
            
                (
                
                    Presentation
                    
                
                )
            > SlidesLive Video | 🔗 | 
| 
            
                Thu 7:10 a.m. - 7:30 a.m.
            
         | Morning Break | 🔗 | 
| 
            
                Thu 7:30 a.m. - 8:15 a.m.
            
         | ML Workloads in AR/VR and Their Implication to ML System Design - Hyoukjun Kwon (UC Irvine)
        
        
        
            
                (
                
                    Invited Talk
                    
                
                )
            > SlidesLive Video | 🔗 | 
| 
            
                Thu 8:15 a.m. - 9:00 a.m.
            
         | Benchmarks for Developing Generalist Agents: Closing the Gap in Real-World Autonomous Decision-Making - Vijay Janapa Reddi (Harvard)
        
        
        
            
                (
                
                    Invited Talk
                    
                
                )
            > SlidesLive Video | 🔗 | 
| 
            
                Thu 9:00 a.m. - 11:00 a.m.
            
         | Lunch | 🔗 | 
| 
            
                Thu 11:00 a.m. - 11:45 a.m.
            
         | Towards AI Workflow Benchmarking with Consideration of Energy Efficiency on Leadership Computing Platforms - Wes Brewer (Oak Ridge National Laboratory)
        
        
        
            
                (
                
                    Invited Talk
                    
                
                )
            > SlidesLive Video | 🔗 | 
| 
            
                Thu 11:45 a.m. - 12:30 p.m.
            
         | Accelerating LLMs with Speculative Inference and Token Tree Verification - Zhihao Jia (Carnegie Mellon University)
        
        
        
            
                (
                
                    Invited Talk
                    
                
                )
            > SlidesLive Video | 🔗 | 
| 
            
                Thu 12:30 p.m. - 12:50 p.m.
            
         | Afternoon Break | 🔗 | 
| 
            
                Thu 12:50 p.m. - 1:50 p.m.
            
         | Panel Session - Wes Brewer (ORNL), Martin Foltin (HPE), Zhihao Jia (CMU), Hyoukjun Kwon (UCI), Yingyan Lin (GA Tech)
        
        
        
            
                (
                
                    Discussion Panel
                    
                
                )
            > SlidesLive Video | 🔗 | 
| 
            
                Thu 1:50 p.m. - 2:00 p.m.
            
         | Conclusion
        
        
        
            
                (
                
                    Closing Remarks
                    
                
                )
            > | 🔗 |