AI4All Day 8: All about Metrics, Self-supervised learning, AI for Social Good

Nidhi Parthasarathy, Thursday, July 7th 2022

The Introduction to the Day

We started the day in our cohort groups. We learned about different metrics in AI. We started by reviewing the ML pipeline development process, and then discussed classification models evaluation metrics (which took up most of the lecture) while highlighting the challenges of poor models. Finally, we were introduced to the MedMNIST dataset that we will be using for our projects. We started with the ML pipeline development process and talked about how we go from code to workflow.

All About Metrics

Then we talked about the metrics. We first went over what a confusion matrix was and got an example of a matrix for cat classification (picture below).

Using this, Alaa explained what each metric was. She explained that accuracy was the number of correct predictions that the model can make out of the total number of predictions. She also talked about precision (true positives over all positives), recall (true positive over true positives plus false negatives), and the F1 score (harmonic mean of precision and recall), their uses and how to represent these with the confusion matrix. Finally, she talked about sensitivity (similar to recall) and specificity (true negative rate). We also learned about ROC (receiving operating characteristic) curves, and AUC (area under the curve) — both popular metrics and graphics used in AI programming.

Next, she talked about some challenges with bad models. For example, she discussed how there are challenges due to datasets not being diverse or from them not addressing different contextual factors that accent modern medicine.

After a short break, we came back and went over the classification model that we created on Wednesday. This portion was confusing for many of the students so it was really helpful to go over it again!

Self Supervised Learning for the Real World

We then had a lunch-time lecture from Alex Tamkin on “Self Supervised Learning for the Real World.” He started by talking about human robot interactions. Next, he talked about unsupervised machine learning and how it goes from unlabeled data to pretraining objectives to model architecture to the transfer method to downstream tasks. He also talked about how, in self-supervised learning, we go from unlabeled data to a generalist model.

One thing that I found particularly interesting in his lecture was the discussion around “How do you use the right features for a desired task?”

He explained how self supervised learning has many successes but some pitfalls (e.g., missing insights) as well. He also talked about NLP vision and speech. He concluded by pointing out that universal SSL provides the foundation for anyone to do good science and linked his github page for more resources.

A Stanford Panel

Next, we had panelists from Stanford come over to talk about their experience in AI. The panelists were Ali Mottaghi, Roshni Sahoo, and Shyamoli Sanghi. I really enjoyed learning about the panelists’ experience, in particular how they found answering societal questions with ML to be very fulfilling. I learnt about opportunities in healthcare, policy, and education.

One notable piece of advice was to find people who are interested in what you are doing and bring something new (a different background or skill set) to the table when working with them.

The panel also discussed how linear algebra was an important foundation for AI. There was also a good discussion of bias in ML, particularly around its application for education and also some of the perils with AI technology not being interpreted as we want. The responsibility falls on AI scientists to make sure it is used ethically. Overall, a really interesting panel that I learnt a lot from.

Continue to the blog post on day 9.