TMLS2021 Workshop: DynaTask, A New Open Source Approach for AI Benchmarking

Speaker: Sam Lightstone, Software Engineer, Facebook

Abstract:

In this session we'll introduce and demo a new open source paradigm for AI benchmarking, called DynaTask. This novel benchmark suite uses dynamic adversarial data collection to evaluate AI models, and assess how easily an AI can be fooled by humans.

What You'll Learn:

• Undestanding AI benchmarking
• Introduction to using DynaBench
• Overview of dynamic adversarial data collection
• Demo of the AI benchmarking software
• How to use and contribute to the open source

Bio:

Sam Lightstone recently joined the technical leadership at Facebook. From 2020-2021 Sam was IBM Chief Technology Officer for AI. From 2017-2020 Sam was the IBM Chief Technology Officer for Data focusing on IBM’s database and big data portfolio. He is cofounder of the IEEE Workgroup on Self-Managing Database Systems. Sam has more than 65 patents issued and pending and has authored 4 books and over 30 papers. Sam’s books have been translated into Chinese, Japanese and Spanish. In his spare time he is an avid guitar player and fencer. His Twitter handle is "samlightstone".