Collaborative Lesson Development Training

# Collaborative Lesson Development Training :::info **Dates**: 17-20 February 2025 **Time**: 17:00 - 21:00 UTC ([see in your local time](https://www.timeanddate.com/worldclock/fixedtime.html?msg=Collaborative+Lesson+Development+Training+Part+1&iso=20250217T17&p1=%3A&ah=4)) **Zoom link**: https://carpentries.zoom.us/j/84502202694?pwd=lPw4b7DPszdvs03epnxp6ya8o8OtKX.1 **Code of Conduct**: https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html **Curriculum**: https://carpentries.github.io/lesson-development-training/ **Trainers:** - Toby Hodges - Sarah Stevens ::: [TOC]  ## About This CodiMD We will use this CodiMD to take notes, share links, exercises, etc with participants throughout the training. Participants are encouraged to take shared notes on this page. The Trainers will show you how to use CodiMD at the beginning of the workshop. ## Attending **Name / pronouns (optional) / affiliation / email address** - Hannah Cohoon / she-her-hers / LBNL / hcohoon@lbl.gov - Rajshree Deshmukh (she/her) LBNL / rajshreed@lbl.gov - Sarah Stevens (she/her/hers) / UW-Madison / sarah.stevens@wisc.edu - Georgia Bullen (she/her/hers) / Superbloom Design / georgia@superbloom.design - Eriol Fox / they/them / Superbloom Design / eriol@superbloom.design - Anh Le / any / Superbloom Design / anh@superbloom.design - Philliph Drummond / name/he/they / Superbloom Design / philliph@superbloom.design - ## Notes Notes archive: https://codimd.carpentries.org/yeQ0JbLZTXK99yKSt7yGHQ?edit ### Designing Exercises **Objectives**: After completing this episode, participants should be able to... - Describe the importance of regular assessment while a lesson is being taught. - Choose the format for an exercise based on the outcome it is intended to measure. **Questions**: - How can you measure learners' progress towards your lesson objectives? - Why are exercises so important in a lesson? - What are some different types of exercises, and when should they be used? - Why should we create assessments before we have written the explanatory content of our lesson? #### Formative Assessments in this Training (5 minutes) Think back through the parts of this training you have followed so far. Identify two examples of formative assessment that the Trainers have employed so far. As an extra challenge, try to decide whether these assessments were used to assess progress towards a particular learning objective and, if so, what the relevant objective might have been. * Rajshree: There were multiple exercises throughout to make us implement what we are trying to learn, and that helped to have deeper conversations aiding further learning by doing. Ex: Defining objectives for lessons and making them SMART objectives helped us to think better on overall our lesson design and goals. * Philliph: First would maybe would be when we had to analyze sample learning objectives to check if they were SMART; second could be when we were first asked to reflect on the best lesson we've experienced and try to identify strengths in both the content and the teaching methods used, then discuss them together. Both of these assessments allowed the trainers to identify gaps in understanding and potentially adjust their instruction accordingly. * Hannah: evaluating learning objectives in the table (learning objective: Evaluate a written lesson objective according to SMART criteria ) & explaining why we chose a dataset (learning objectives: find candidate datasets and evaluate their suitability ) * Georgia: Learning & Discussion > Applying through exercise and reflecting and then adapting as a flow of the training, e.g. the lesson objectives, episode objectives, SMART measurements. * Eriol: The SMART table and then leading to producing a combo of objective + SMART objectives for the episode we are solely responsible for. I expect the object was broadly: Intro SMART objectives, assess whether learners can distinguish between existing examples what parts are SMART and then can they (without examples) apply this to their lesson development context via episode creation objectives. * Anh: 1- Example learning objectives table: assessing our understanding of SMART learning objectives; 2 - does this exercise count? jk the times where we're developing our own learning objects for the lesson & episodes: applying prev knowledge to build upon the learning objective concept #### Recommended reading: More Example Assessment Types * [Exercise Types Chapter from Teaching Tech Together](https://teachtogether.tech/en/index.html#s:exercises) * [Edutopia’s 56 Examples of Formative Assessment](https://www.edutopia.org/groups/assessment/250941) * ["21 Ways to Check for Student Understanding"](https://www.opencolleges.edu.au/informed/features/21-ways-to-check-for-student-understanding/) * [H5P Examples and Downloads for Interactive Content](https://h5p.org/content-types-and-applications) #### Exercise: Exercise Types and When to Use Them (15 minutes) The Trainers will assign you to pairs or small groups, and give each group an exercise type to work on. Each group should assign a notetaker, to summarise your discussion at the end of the exercise. ##### For lessons predominantly teaching non-coding skills and concepts The Trainers will assign your group a type of exercise to focus on. Read about your given exercise type on the indicated pages of [_Is This a Trick Question?_](https://cdm16884.contentdm.oclc.org/digital/collection/p16884coll42/id/147): * multiple choice (page 13) * true-false (pages 20 & 21) - Ahn, Georgia, Hannah * fill-in-the-blank (page 34) * authentic assessment (pages 46 & 47) - Eriol, Philliph, Rajshree Then, discuss the following questions together: * Could exercises of this type be used in your lesson? * If so, can you identify any of your written objectives that could be assessed with an exercise of this type? * Have your Trainers set you any exercises of this type in this training so far? * Would this type of exercise be suited to a novice audience? Or is it a better fit for intermediate or advanced learners? * Would this kind of exercise work well in an in-person workshop setting? Would it be better suited to self-directed learning and/or a virtual workshop? ##### True/False - Ahn, Georgia, Hannah * Could exercises of this type be used in your lesson? * Yes! * If so, can you identify any of your written objectives that could be assessed with an exercise of this type? * Yes > Identify Participation, e.g. "[Blank] is a risk — T/F" * Feels useful to use the forced choice variation, more than the T/F format * Weaker action verbs seem like they'd need questions more than exercises (e.g. "*determine* when data collection can end"). * Using extremes to prompt misconceptions * Concern: Prompts that lead to remembering the wrong answer >> being selective about when to use T/F and multiple choice * Have your Trainers set you any exercises of this type in this training so far? * Learning Objectives assessment exercise (the table)? >> challenge of "maybe", when something isn't fully false or fully true * Would this type of exercise be suited to a novice audience? Or is it a better fit for intermediate or advanced learners? * the format isn't about level of the learner, it's more about matching to the content and exercise, and matching to the level of the audience. * b/c of the concern around teaching people the wrong thing, inclined to lean expert level over novice * Would this kind of exercise work well in an in-person workshop setting? Would it be better suited to self-directed learning and/or a virtual workshop? * All of the above. * Fosters discussion in-person (or sync virtual), also good in a tool * If there's grey area, not great for self-directed learning ##### Authentic Assessment - Eriol, Philliph, Rajshree * Could exercises of this type be used in your lesson? * Yes < agreed! * What the learners wrote first and what they wrote last and whats the reflection on the learning path = boiling frog mode * If so, can you identify any of your written objectives that could be assessed with an exercise of this type? * Objective generally = confidence in your learning, progress, pathway and results. (not the 1+1=2 but the 'eyeballing it' measurement method) * Meta level curriculum = asking more and better questions, giving and recieving feedback, thinking about feedback = the 5 whys (method) * Have your Trainers set you any exercises of this type in this training so far? * Yes * conduct social-science research! * write stories and reports! * read and interpret literature * ^^ but how to tell how well * Would this type of exercise be suited to a novice audience? Or is it a better fit for intermediate or advanced learners? * Depends on the attitudes of any learner * Would this kind of exercise work well in an in-person workshop setting? Would it be better suited to self-directed learning and/or a virtual workshop? * Yes especially if peer-based review and learning is possible. (burden on the instructor experts could be high) The exercise types chapter of _Teaching Tech Together_ https://teachtogether.tech/en/index.html#s:exercises Resume at XX:30 #### Exercise: Assessing Progress Towards an Objective (30 minutes) Using one of the exercise formats you have learned about so far or another one you can read about in the resources shared, design an exercise that will require learners to perform one of the actions described in the objectives you wrote earlier, and that assesses their ability to do so. Draft the exercise in your shared lesson design notes document. These should be assessments of the lower-level objectives defined for individual episodes in the lesson, as opposed to the lesson-level objectives you wrote first. Trainees working as a team can choose whether to work together on discussing and designing a single exercise to assess a single objective, or to divide their efforts and each focus on an exercise for their own episode. If you choose to take the latter approach and finish with time to spare, spend the remainder reviewing and providing feedback on one another's assessments. **Key Points**: - Assessments are a way to determine whether the objectives you defined for the lesson have been reached. - Exercises help learners commit what they've learned to long-term memory. - Some types of exercises are better for particular audiences and to address certain objectives. - **Formative assessment** happens *during teaching* and provides feedback both to an instructor and a learner - about progress and whether learning of new concepts occurred but also about any misunderstandings and misconceptions which can hinder further learning. --- ### How to Write a Lesson **Objectives**: After completing this episode, participants should be able to... - Estimate the time required to teach a lesson. - Summarise the content of a lesson as a set of questions and key points. - Connect the examples and exercises in a lesson with its learning objectives. **Questions**: - Why do we write explanatory content last? - How can I avoid demotivating learners? - How can I prioritise what to keep and what to cut when a lesson is too long? #### Exercise: Examples Before Exercises (20 minutes) Looking back at one of the exercises you designed before: what examples could you include in your narrative to teach learners the skills they will need to apply to complete the formative assessments you have designed? Outline one of these examples in your design notes document #### Lesson Time Management (10 minutes) (5 minutes) In the shared notes document, note down your answers to these questions: - From a design perspective, at what point is a lesson too long? - What factors influence and constrain the length of a lesson? - How might you prioritise what to keep if you have to cut lesson content down? * Eriol: * When you loose track of the/a single point (objective) of a lesson so when the content doesn't consistently circle back on the clear learning nugget. When there are more than one (or any) tangents or pre-requesite knowledge/understanding to review * Influence/Constrain = the variables like lesson narrative becoming convoluted, if people are accessing too much of their long term and short term memory simulataneously * Cut = long reads without demos/engaging activities (better to do a task quickly/messily and attempt it than never get to a task/exercise). Content that doesn't allow learners to express themselves and their process/progress. * Georgia: * Too long — [signs during] people feeling overwhemled/lost, not being able to focus/stay; people needing to re-confirm the goal/purpose frequently; * Factors: considering what people (trainers and participants) can commit to, finding an amount of time that allows people to not get too into perfection, but stay in a more active/exploratory learning space * How to prioritize: picking 1-2 key things that achieve the objective, vs including all the things * Philliph: * when is lesson too long * when learners tell/show you that its too long * when there is considerable overlap or collision with the preceeding or following lesson * what factors influence or constrain * course size, number of trainers, # of participants and their engagement * how might i prioritize * refer to feedback from other trainers/instructors or learners * refer back to overall and individual lesson objectives * considering if there duplicate/redundant content that is being shared across lessons (might need to break-up/sprinkle) * Anh: * If there's too much material and participants feel overwhelmed, if the participants start skipping ahead or skipping content, if participants get stuck; timewise, 3 full days is a lot * Day jobs and other life responsibilities; if the learner is seeing how the lessons can apply to their current problem/use cases * Prioritizing exercises or episodes that best communicate the learning objectives and marking others as supplementary; identifying core materials that can get people up & running & excited in the initial stage so they'll come back for practice or more learning * Hannah: * too long: can't get sign ups, can't maintain engagement, doesn't focus on a coherent and standalone topic, can't have an instructor commit to the time, * influential factors: objectives, delivery format, participants' expectations for how long it takes to learn something useful, number of potential instructors (are there 5 people in the world who can do it? or 500?), * priorities: can you cut this and still meet the learning objectives? does it need an exercise? is there an external resource that you can point people to instead? * Rajshree: * Lesson too long when: * Participants stop absorbing content properly - would know from their output in exercises * too much saying and less interactive exercises * instructors start to fall back or miss covering certain topics/ exercises in lessons * Factors influencing the length: * number of episodes & sections (different topics to cover) * number of exercises and breaks required to digest the topics * exercise intensitiy & duration required to do, assess and discuss them * How to prioritize what to keep: * if episode objectives are not directly impacting the main lesson objective - classify episodes and sections into primary and secondary in terms of importance to main lesson objectives. * evaluate if exercises can be changed to become shorter/ faster * or cut short exercises for second priority topics Eriol: wht are the longest and shortest lessons that the Carpentries has? a good, fairly short one: https://librarycarpentry.github.io/lc-computational-thinking/index.html a long, two-day curriculum: https://datacarpentry.github.io/image-processing/ #### Review Your Text for Demotivations - dismissive language - e.g. ‘simply’, ‘just’ - use of stereotypes - check learner profiles for stereotypes too - expert awareness gaps, i.e. places where you may be assuming the learners know more than they actually do - fluid representations, i.e. using different terms with the same meaning interchangeably - unexplained or unnecessary jargon/terminology - unexplained assumptions - sudden jumps in difficulty/complexity #### Review Your Text for Accessibliity - Avoiding regional/cultural references and idioms that will not translate across borders/cultures - Avoiding contractions i.e. don’t, can’t, won’t etc. - Checking that all figures/images have well written alternative text, including writing altnerative text for data visualizations. - Checking the header hierarchy - no h1 headers in the lesson body, no skipped levels - Using descriptive link text - no “click here” or “this page”, etc. - Checking the text and foreground contrast for images [Glosario](https://glosario.carpentries.org/), a multi-lingual glossary of data science terms curated by The Carpentries community. #### Exercise: Explain Your Terminology (5 minutes) In your shared notes document, add a list of terms or jargon from your lesson, along with their definitions. You do not have to have a complete list now, but it is good to start working on it -- you can always come back and update it later. design (in this specific context) design decision session user scenario journey usability interview protocol user qualitative quantitative human-centered stakeholder method product user input research meathod stakeholders feature GUI UI UX CLI privacy interpret (the) 'experience' usability assessment clustering transcribe (success) metric IRB screener user needs grouping hypotheses insights informed consent outreach plan synthesis **Key Points**: - The objectives and assessments provide a good outline for an episode and then the text fills in the gaps to support anyone learning or teaching from the lesson. - It is important to review your lesson for demotivating language, cognitive load, and accessibility. - To reduce cognitive load and ensure there is enough time for for the materials, consider which lesson objectives are not needed and remove related content and assessments. #### Exercise: Completing episode metadata (10 minutes) In your shared design notes document, add key points and questions to your episode. ### The Carpentries Workbench **Objectives**: After completing this episode, participants should be able to... - Identify the key tools used in The Carpentries lesson infrastructure. - Complete the fundamental setup steps for a new lesson repository. - Edit Markdown files using the GitHub web interface. **Questions**: - How is a lesson site set up and published with GitHub? - What are the essential configuration steps for a new lesson? - How do you create and modify the pages in a lesson? #### Creating a lesson repository Templates for new lesson repositories: * [For lessons using R Markdown](https://github.com/carpentries/workbench-template-rmd/generate) * recommended for lessons that will contain example R code blocks/programs * [For lessons using Markdown](https://github.com/carpentries/workbench-template-md/generate) #### Improving the `README.md` (5 minutes) Take a few minutes to update it with some basic information about the project: - the lesson title - a short description of the lesson - a list of the names of the authors, optionally linked to their GitHub profile **Key Points**: - Lesson sites are built from source repositories with GitHub Pages. - A new lesson repository can be created from a template maintained by The Carpentries, and configured by adjusting the `config.yaml` file. - The main pages of a lesson website are created from Markdown or RMarkdown files in the `episodes` folder. --- ### Adding Lesson Content **Objectives**: After completing this episode, participants should be able to... - Add lesson episodes as individual pages of a lesson website. - Use _fenced divs_ to create different structural elements within a page to format objectives, questions, keypoints, exercises and their solutions in a lesson website. **Questions**: - How do you create and modify the pages in a lesson? - How should different structural elements be presented in a lesson website? #### Exercise: practice editing Markdown in GitHub Add the objectives you defined for your lesson as a bullet list in the `index.md` file of your lesson repository. #### Exercise: Formatting Exercises in a Lesson Site (15 minutes) Using the approach demonstrated above, format the exercise you designed previously as an exercise in your lesson site. [The Workbench Component Guide](https://carpentries.github.io/sandpaper-docs/component-guide.html) includes an example of each kind of fenced div that you can use in your lesson. The Workbench documentation provides [instructions for installing the infrastructure on your local system](https://carpentries.github.io/sandpaper-docs/index.html#setup) - This can be really helpful and save you a lot of time, in comparison to always having to commit your changes and wait for the GitHub Actions build process to complete before you can see how the new version looks on the rendered site. #### Exercise: More Practice (15 minutes) Use this time to add more content to your lesson site. Generally speaking, you should try to transfer the drafted content from your lesson design notes into the lesson website. Here are some suggestions for things you might try: 1. Add your list of prerequisite knowledge/skills as a fenced div with the `prereq` class to your `index.md` file. 2. [Add some tabbed content](https://carpentries.github.io/sandpaper-docs/episodes.html#tabbed-content) in your episode. 3. Start a glossary of terms in the `learners/reference.md` file, referring to [the Workbench documentation on how to add a list of term definitions](https://carpentries.github.io/sandpaper-docs/editing.html#learners). 4. Look through the [Workbench component guide](https://carpentries.github.io/sandpaper-docs/instructor/component-guide.html) and try implementing some of the other flavours of fenced div. 5. Add a new Markdown file to the `learners/` or `instructors/` folder and see if you can find the built page in your lesson site. Instructions for configuring automated pull requests to keep your workflow files up to date: https://docs.carpentries.org/resources/curriculum/lesson-forks.html#configure-maintenance-workflows ### How we Operate **Objectives**: After completing this episode, participants should be able to... - Describe the role that feedback plays in the life cycle of a lesson. - Connect with other members of the community. **Questions**: - What are the important milestones in the development of a new lesson? - How can The Carpentries lesson development community help me complete my lesson? #### Lesson Life Cycle Revisited ![The life cycle of a lesson, annotated to indicate the platforms provided for lesson projects at each stage of the cycle.](https://carpentries.github.io/lesson-development-training/fig/life_cycle_annotated.png) - `pre-alpha`: first draft of lesson is being created - `alpha`: lesson is being taught by authors - `beta`: lesson is ready to be taught by other instructors - `stable`: lesson has been tested by other instructors and improved based on feedback. Major changes and updates are relatively infrequent. #### Resources for Lesson Developers * Pilot observation notes template: https://codimd.carpentries.org/lesson-pilot-observation-notes-template# * Handbook page about pilot workshops: https://docs.carpentries.org/resources/curriculum/lesson-pilots.html * Incubator Proposals repository: https://github.com/carpentries-incubator/proposals * The Lesson Developer Handbook: https://docs.carpentries.org/handbooks/lesson_developers.html **Key Points**: - Teaching a lesson for the first time is an essential intermediate step in the lesson development process. - The Carpentries lesson developer community shares their experience on multiple communication channels. --- ### Preparing to Teach **Objectives**: After completing this episode, participants should be able to... - Summarise lesson content as a teaching plan. - Add Setup Instructions and Instructor Notes to the lesson site. - Create a feedback collection plan. **Questions**: - What can I do to prepare to teach my lesson for the first time? - How should I communicate lesson setup instructions to learners? - What information should be recorded for instructors teaching a lesson? - How should information be collected as part of the feedback process? [the Virtual Minute Card template](https://docs.google.com/forms/d/1p7iOV5HNvy4POS4g6eottY8RSfKq4kaoKz1-jIFYTMI/template/preview) Training Pilot Operations Guide: https://doi.org/10.5281/zenodo.14066425 #### Exercise: Homework The final part of this training will focus on the skills needed to collaborate effectively. Before that there will be a break, during which we would like you to complete the following three tasks: 1. Teach one episode of your lesson (probably the one you have been working on in these two days). See the [Lesson Trial Runs](https://carpentries.github.io/lesson-development-training/trial-runs.html) page for full details. 2. After your trial run has concluded (immediately after, or when you have reviewed any feedback you collected from learners), note down your answers to the following questions: - What worked? - What did not? - What will you do differently next time? - What will you change in your material you taught? We will refer to these notes when we reconvene for the last episode of this training. 3. Based on your experience teaching the material and the feedback you received from your learners and helpers, make a list of issues you have identified with the material you prepared, e.g.: - examples that did not work as expected, - improvements that could be made to exercises, - parts that learners found particularly challenging, - unexpected questions or misconceptions that came up during the trial run. We will return to these notes during the final training session, so please make sure you save them somewhere you will be able to find them again easily when the time comes. #### Polish a Lesson (20 minutes) This exercise provides you with a chance to look back over everything you have sketched out for your episode(s) and the lesson as a whole and consider what still needs to be done before it can be taught. You can use this time however you judge will be most beneficial to your preparations for teaching your episode in a trial run. If you are not sure how to start, consider the following prompts: - Which of the exercises set so far in this training did you not have time to complete? - What do you still need to add/work on? - What can you remove/consider removing? - How will the narrative and example data you have chosen for your lesson support teaching and assessment? - What diagram or other visual aids could you create/add to supplement your text? #### Organise Your Knowledge (10-15 min) Take some time to think back on what has been covered so far, then make some notes on the most important points and actions you want to take away from that. The Trainers and other participants will not look at this - it is only for you. If you do not know where to start, consider the following list for a starting point: - draw a concept map, connecting the material - draw pictures or a comic depicting one or more of the concepts - write an outline of the topics we covered - write a paragraph or “journal” entry about your experience of the training today - write down one thing that struck you the most #### Optional Homework: Assessing Your Lesson Design (This exercise adapted from [Via et al. 2020](learners/reference.md).) When the first draft of your lesson content is nearly complete, consider mapping out the relationships between the objectives of your episode and the examples and exercises via which they will be taught and assessed. For example, > The read CSV and inspect demo supports Objective 2 > (load a simple CSV data set with Pandas) > and will be delivered using participatory live coding. > The objective will be assessed with an exercise that > requires learners to apply the read_csv function to another file > and count the rows in the resulting DataFrame object. - Does any of your planned content not support any learning objectives? - Is there at least one piece of content planned for each learning objective? - Is there a formative assessment planned for each learning objective? Please give us feedback again! https://forms.gle/4RW98dsAUqEoWZZH