{"id":39,"date":"2024-02-23T15:12:25","date_gmt":"2024-02-23T15:12:25","guid":{"rendered":"https:\/\/responsiblepraxis.ai\/?page_id=39"},"modified":"2024-04-13T21:34:32","modified_gmt":"2024-04-13T21:34:32","slug":"lesson-plan-ai-audit","status":"publish","type":"page","link":"https:\/\/responsiblepraxis.ai\/?page_id=39","title":{"rendered":"AI Audit Lesson Plan"},"content":{"rendered":"\n<p>AI Audit Lesson Plan [80 minute class period]\u00a0Lesson Plan AI Audit<\/p>\n\n\n\n<p>The purpose of this lesson plan is to equip students with the skills and background knowledge to perform an AI audit. It accomplishes this by walking through the process of an AI audit on a fictional AI.\u00a0<\/p>\n\n\n\n<p>This lesson plan is informed by<\/p>\n\n\n\n<p> \u201c<a href=\"https:\/\/www.google.com\/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwi2z8GgtMKEAxUaiO4BHTRcB5cQFnoECCAQAQ&amp;url=https%3A%2F%2Facademiccommons.columbia.edu%2Fdoi%2F10.7916%2FD8TT536K%2Fdownload&amp;usg=AOvVaw3V5G9CkxP4-6Hh69VQUdTW&amp;opi=89978449\">Algorithmic Accountability<\/a>\u201d by Nicholas Diakopoulos<\/p>\n\n\n\n<p>and <\/p>\n\n\n\n<p>&#8220;<a href=\"https:\/\/websites.umich.edu\/~csandvig\/research\/Auditing%20Algorithms%20--%20Sandvig%20--%20ICA%202014%20Data%20and%20Discrimination%20Preconference.pdf\">Auditing Algorithms<\/a>&#8221; by Christian Sandvig et al. <\/p>\n\n\n\n<p><\/p>\n\n\n\n<p>Play&nbsp;<a href=\"https:\/\/www.survivalofthebestfit.com\/resources\" target=\"_blank\" rel=\"noreferrer noopener\">Survival of the Best Fit<\/a>&nbsp;[10 min] followed by discussion [5 min]<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>What did you notice in the game?&nbsp;&nbsp;Did anything surprise you?&nbsp;What areas would you immediately want to investigate about this AI?<\/li>\n\n\n\n<li>Is there anything from this game that you think is applicable to real-world AI?&nbsp;<\/li>\n<\/ul>\n\n\n\n<p>Connect to the real world. This is a fictional AI, but stuff like this really happens! These are just a few examples [5 min]<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Amazon resume screening&nbsp;&nbsp;<\/li>\n\n\n\n<li>Predictive Policing&nbsp;<\/li>\n\n\n\n<li>ProPublica Risk Assessment&nbsp;<\/li>\n\n\n\n<li>Healthcare&nbsp;<\/li>\n\n\n\n<li>Facial Recognition&nbsp;<\/li>\n<\/ul>\n\n\n\n<p>We know about these AI fails because of journalistic AI audits. Given that companies don\u2019t share their training data and algorithms, one of the most useful tools thus far is the AI audits. The goal of this lesson is to equip you with the skills to do it yourself. [5 min]&nbsp;<\/p>\n\n\n\n<p>How to perform an AI Audit [50 min]&nbsp;&nbsp;<\/p>\n\n\n\n<p>We\u2019re going to practice an AI audit in three steps: 1) Identify an AI to audit, 2) Sample the AI, and 3) Find the story. We will run through these steps with the fictional AI.&nbsp;<\/p>\n\n\n\n<p>Identify an AI to Audit.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>We\u2019ve already done that in this case, but how can we tell that this is an interesting audit? (It has real-world consequences that are newsworthy and important, it hasn\u2019t been audited before, etc.)<\/li>\n\n\n\n<li>How would you go about finding an AI to audit in a real AI audit? (think about systems you interact with, gatekeeping metrics like credit scores, think about any time some complex aspect of human life is quantified, software that prioritizes, classifies, or filters).&nbsp;<\/li>\n\n\n\n<li>How would you tell an AI wouldn\u2019t be interesting to audit? (While almost any AI could be potentially interesting, AI that has very low stakes is harder to make a case for, so are things that have been extensively studied. There is a lot of work on YouTube\u2019s algorithm but little on TikTok\u2019s.)<\/li>\n<\/ul>\n\n\n\n<p>Sample the AI.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Imagine we\u2019re external to the company in the game. This company is selling their hiring AI to your company. What information would you want before you would make the decision to purchase their AI service?&nbsp;<ul><li>Be sure to cover 1) the data you\u2019d need to make sure it actually does the job they say it does\u2014maybe the rates of people it recommended hiring getting let go within 2 years, 2) the data you\u2019d need to tell if it is biased against protected groups\u2014maybe the hiring rates of different demographics before and after a different company implemented the algorithm, and anything else students think of.<\/li><\/ul>\n<ul class=\"wp-block-list\">\n<li>This is a kind of \u201cblue sky\u201d audit. You will most likely not have access to the information you\u2019d need to truly audit the AI, but it\u2019s an important step to clarify what your goals are and what you\u2019d like to check for. It\u2019s also a good indication of the kind of data AI companies should make public so that they can be audited. A blue sky exercise can also help students figure out alternate ways they could get at similar information.&nbsp;<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>What makes sampling an AI hard in the real world?<ul><li>Blackbox nature of AI. Internal system inaccessible\u2014only inputs and outputs.<\/li><\/ul><ul><li>Additionally, many times you don\u2019t even have access to the inputs, only outputs.<\/li><\/ul><ul><li>Many AIs are tailored to you. It\u2019s hard to get a representative sample.<\/li><\/ul>\n<ul class=\"wp-block-list\">\n<li>It\u2019s difficult to show causation. Correlation is easier.&nbsp;<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>Find the story.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>What would your story be for our example AI?&nbsp;&nbsp;<\/li>\n\n\n\n<li>Prompts to help in coming up with a story:&nbsp;<ul><li>Has the algorithm made a bad decision or broken an expectation for how we think it should be operating?&nbsp;<\/li><\/ul><ul><li>How does it affect individuals and or communities<\/li><\/ul><ul><li>Are there any false positives or false negatives?&nbsp;<\/li><\/ul>\n<ul class=\"wp-block-list\">\n<li>What is driving that break in expectations?\n<ul class=\"wp-block-list\">\n<li>This last question is harder because it requires knowing more about the AI\u2019s inner workings, but if you can dig into it, you really have a story.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>Get into groups and search for AIs to audit. Come together and discuss what you found and ways to audit. [10 minutes]&nbsp;<\/p>\n\n\n\n<p>Questions&nbsp;and create individual plans for sampling an AI [10 minutes]&nbsp;<\/p>\n\n\n\n<p>Student AI audit examples available <a href=\"https:\/\/responsiblepraxis.ai\/?page_id=9\">here<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>AI Audit Lesson Plan [80 minute class period]\u00a0Lesson Plan AI Audit The purpose of this lesson plan is to equip students with the skills and background knowledge to perform an AI audit. It accomplishes this by walking through the process of an AI audit on a fictional AI.\u00a0 This lesson plan is informed by \u201cAlgorithmic [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":235,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-39","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/responsiblepraxis.ai\/index.php?rest_route=\/wp\/v2\/pages\/39","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/responsiblepraxis.ai\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/responsiblepraxis.ai\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/responsiblepraxis.ai\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/responsiblepraxis.ai\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=39"}],"version-history":[{"count":5,"href":"https:\/\/responsiblepraxis.ai\/index.php?rest_route=\/wp\/v2\/pages\/39\/revisions"}],"predecessor-version":[{"id":51,"href":"https:\/\/responsiblepraxis.ai\/index.php?rest_route=\/wp\/v2\/pages\/39\/revisions\/51"}],"up":[{"embeddable":true,"href":"https:\/\/responsiblepraxis.ai\/index.php?rest_route=\/wp\/v2\/pages\/235"}],"wp:attachment":[{"href":"https:\/\/responsiblepraxis.ai\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=39"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}