NOTE: This is a special session for a couple of reasons:

1) We're back after a hiatus!

2) The feedback from this session will be integrated into the research project that we have going on at MAIEI this summer where we are joined by interns from across the world working on this subject! For more information, check out: https://montrealethics.ai/meet-the-16-inaugural-maiei-summer-research-interns/

Stradigi AI is hosting the Montreal AI Ethics Institute and the local AI ethics community at their offices to discuss the very important subject of reproducibility in machine learning which is causing a major crisis in the field, especially as it leads to wasted time and effort in trying to utilize published results but not being able to obtain results because of a lack of published code, hyperparameter information, data sources, network structure, etc.

In this session we'll be looking to gain a holistic understanding by leveraging insights from a diversity of backgrounds and fields, both from a social science and technical perspective. We'll be building on the work from the Research Internship Program project at MAIEI (material for that will be sent out closer to the session, please make sure to keep an eye out for the email around July 28/29).

Guiding questions for the session:

1) What are the different components apart from capturing models and data sources in a version control system that can help with enhancing reproducibility? (think beyond tools like DVC, cookie-cutter, Pachyderm, MLFlow, etc.) Ideas around solution design, model legacy in terms of choices and long-term maintainability are especially encouraged. Insights from other disciplines might be particularly useful here!

2) How do we make reproducibility more accessible and frictionless for all members who are a part of a ML project including the domains of design, business, executives and of course software engineering and ML/DS? From what we've observed, having high-friction in existing workflows is the biggest reason for non-adoption of such measures to encourage reproducibility.

Mandatory Readings:

0-a) Machine Learning 101 [Very strongly recommended for those without a technical background] https://docs.google.com/presentation/d/1kSuQyW5DTnkVaZEjGYCkfOxvzCqGEFzWBy4e9Uedd9k/edit#slide=id.g168a3288f7_0_58

0-b) AI Ethics 101 [Very strongly recommended for those that are getting started with understanding issues in this space] https://www.youtube.com/watch?v=Z3Tme0WU5D8

Caution that the following readings mostly deal with the technical aspects but don't address the complementary social aspect of this problem - if you find interesting material on that please reach out to abhishek@montrealethics.ai to have the readings be included for the session.

1) Data Science’s Reproducibility Crisis - https://towardsdatascience.com/data-sciences-reproducibility-crisis-b87792d88513

2) Before reproducibility must come preproducibility - https://www.nature.com/articles/d41586-018-05256-0

3) (Deeper dive) Some invited talks from the ICML 2018 workshop on reproducibility in machine learning https://sites.google.com/view/icml-reproducibility-workshop/icml2018/slides?authuser=0

4) (Deeper dive) Accepted papers at the ICLR 2019 workshop on reproducibility in machine learning (most are focused on Reinforcement Learning) https://openreview.net/group?id=ICLR.cc/2019/Workshop/RML

5) Deeper dive material from Xavier at MILA who presented this work at ICML a few weeks ago:

Paper: http://proceedings.mlr.press/v97/bouthillier19a.html
Presentation (at 34mins): https://www.facebook.com/icml.imls/videos/308727963404001/UzpfSTU0NTczMTI4ODoxMDE1NzUwMDA4ODcwNjI4OQ/
Poster: https://postersession.ai/poster/unreproducible-research-is-reproducible/

NOTE : Please join Slack via http://bit.ly/ai-ethics-signup as we will be actively discussing things there leading up to the session.

Also, please make sure to sign up for the Montreal AI Ethics Newsletter https://bit.ly/maieisubscribe because we'll be sharing back the results from the session there.

Format :

5:45-5:55 Registration and networking

5:55-6:00 Introduction

6:00-6:05 Break out into groups

6:05-7:35 Group discussion

7:35-7:45 Synthesize group discussion

7:45-8:10 Group presentations and debate [3 min. presentation + 3 min cross-questioning by group]

8:10-8:15 Session wrap-up