Collaborative Lesson Development Training

Schedule

(All timings listed in UTC)

This schedule is tentative, and exact timings may vary during the event. However, regular breaks will be taken regardless of how the schedule for the specific sections may change.

Day 1

Time (UTC)	Session Title
17:00 - 17:45	Introduction
17:45 - 17:55	Break
17:55 - 18:20	Lesson Design
18:20 - 19:05	Identifying Your Target Audience
19:05 - 19:20	Break
19:20 - 20:10	Defining Lesson Objectives/Outcomes (1)
20:10 - 20:20	Break
20:20 - 21:00	Defining Lesson Objectives/Outcomes (2)
21:00	End

Day 2

Time (UTC)	Session Title
17:00 - 17:30	Episodes
17:30 - 17:45	Break
17:45 - 18:40	The Carpentries Workbench (1)
18:40 - 18:55	Break
18:35 - 19:00	The Carpentries Workbench (2)
19:00 - 19:15	Break
19:15 - 19:50	Defining Episode Objectives
21:00	End

Day 3

Time (UTC)	Session Title
17:00 - 17:55	Designing Assessments (1)
17:55 - 18:10	Break
18:10 - 18:45	Designing Assessments (2)
18:45 - 19:00	Break
19:00 - 19:35	Implementing Exercises
19:35 - 19:50	Example Data and Narrative
21:00	End

Day 4

Time (UTC)	Session Title
17:00 - 17:50	How To Write A Lesson (1)
17:50 - 18:00	Break
18:00 - 18:20	How To Write A Lesson (2)
18:20 - 18:45	How We Operate
18:45 - 19:00	Break
19:00 - 20:05	Preparing to Teach
20:05 - 20:20	Break
20:20 - 21:00	Wrap-up
21:00	End

About This CodiMD

We will use this CodiMD to take notes, share links, exercises, etc with participants throughout the training.

Participants are encouraged to take shared notes on this page. The Trainers will show you how to use CodiMD at the beginning of the workshop.

Attending

• Name (pronouns (optional)) / affiliation / email address
• Toby Hodges (he/him/his) / The Carpentries / tobyhodges@carpentries.org
• Erin Becker (she/her/hers) / The Carpentries / ebecker@carpentries.org
• Sarah Stevens (she/her/hers) / University of Wisconsin-Madison / sarah.stevens@wisc.edu
• Saira Haider (she/her/hers) / US Geological Survey / shaider@usgs.gov
• April Clyburne-Sherin (she/her/hers) / UPenn / aprilcs@seas.upenn.edu
• Kaija Gahm (she/her) / UCLA / kgahm@g.ucla.edu
• Hao Ye (he/him) / UPenn - Community for Rigor / haoye@seas.upenn.edu
• Madison Langseth (she/her) / U.S. Geological Survey / mlangseth@usgs.gov
• Kevin Rusch (he/him) / UPenn (C4R) & UMass Chan / kwrusch@seas.upenn.edu
• Ani Govjian (she/her) / UPenn (C4R) / agovjian@seas.upenn.edu
• Peter Laurin (he/him/his) / UCLA / peterlaurin@ucla.edu
• Karen Word (she/her) / UPenn - Community for RIgor / wordk@seas.upenn.edu
• Xochitl “so-chill” Ortiz Ross (she/they) / UCLA / xortizross@g.ucla.edu
• Eric Martinez (he/him/his) / U.S. Geological Survey / emartinez@usgs.gov

Notes - Day 4

Notes from previous days

Notes from yesterday’s material have been move to an archive document

Feedback from Day 3 (paraphrased and summarized)

Thank you everyone again for your detailed feedback!

Things you learned that you found useful

• Markdown sandpaper syntax and learning the workbench platform
• Introducing a topic and then going into breakout rooms for disucssion is helpful
• Different kinds of exercises for assessments
• The discussion of image/dataset licensing x2
• Time to create exercises and then putting them in our lesson webpages - liked this pattern

Improvement feedback

• Felt quick today
• Confusing when the two people introduce an exercise slightly differently, maybe separate person to introduce exercise isn’t needed
• More time working on our lessons, discussions today not as helpful
• Still struggling with CodiMD x2
• Would have liked to discuss multiple question types in individual groups

Picking up where we left off

Exercise: Choosing a Dataset or Narrative (14 minutes for today, maybe more tomorrow)

Referring to the advice given above, find an appropriate dataset or a narrative for your lesson. Identify one or more potential candidates and note down the advantages and disadvantages of each one.
As a reminder, here are some aspects we suggest that you consider:

• For datasets:
  • size
  • complexity
  • “messiness”/noise
  • relevance to target audience
  • availability
  • license
  • ethics
• For narratives:
  • authenticity
  • relevance to target audience
  • complexity
  • possibility to teach useful things first/early

Takes notes in your Lesson Design Notes document about your discussion and the decisions made. It may be particularly helpful to record:

• Which datasets and narratives did you consider?
• How and why did you choose between them?
• What implications do you think your choice of dataset and/or narrative will have for the design and further implementation of your lesson?

C4R

• descriptive data: Palmer Penguins
• want some other dataset with panel, longitudinal, treatments – allows for interesting decisions on grouping and choices for plots
• want to introduce some errors / outliers
  https://mavenanalytics.io/data-playground

options:

• https://portal.conp.ca/dataset?id=projects/Longitudinal_structural_MRI_and_behavioural_data_for_mice_prenatally_exposed_to_maternal_immune_activation_either_early_or_late_in_gestation
• https://github.com/conpdatasets/MB-Heterogenous-task-representative-data

How to Write a Lesson

Objectives:
After completing this episode, participants should be able to…

• Estimate the time required to teach a lesson.
• Summarise the content of a lesson as a set of questions and key points.
• Connect the examples and exercises in a lesson with its learning objectives.

Questions:

• Why do we write explanatory content last?
• How can I avoid demotivating learners?
• How can I prioritise what to keep and what to cut when a lesson is too long?

Lesson Time Management (10 minutes)

(5 minutes) In the shared notes document, note down your answers to these questions:

• From a design perspective, at what point is a lesson too long?
• What factors influence and constrain the length of a lesson?
• How might you prioritise what to keep if you have to cut lesson content down?

(5 minutes) In the remaining time, your Trainers will lead a discussion based on the responses.

• April:

  • Too long: When it is difficult for a student to summarize the lesson easily. When there are too many objectives. When the students have been doing the “same thing” for a long time.
  • Constraints: Cognitive load - the lesson should end before the students “need” a break.
  • Prioritise: Feel it out with students - you can see them start to fidget or glaze over. Run it for a colleague - see if they can help you identify the key things and drop things you are attached to but aren’t necessary.

• Ani:

  • Too Long:In terms of visual design, when the text gets unweidly and looks more like a stream of text without a break. If there are breaks, variations, pauses for activities, the overall length (visually) is not a problem. As far as demands on student attention, 15-20 min is a useful starting place for most new content with short breaks. Stringing a few of these 15-20min chunks together is great, but probably not exceeding 2 hours for most intensive content.
  • Length constraints: Schedules, attention span, difficulty of topic.
  • Prioritization: Most often, I think it’s best to offer foundations for learners to build on, but you can also start with some intro and then invest heavily in “the most important” item.

• Eric:

  • Too long:
    • Running over on allocated time
    • Feedback during alpha/beta testing
  • Constraints:
    • Relevance to lesson learning objectives
    • Available time
    • Cognitive burnout
  • Priorities:
    • Focusing on what is both necessary and sufficient to achieve desired learning objectives

• Hao:

  • rule of thumb: a lesson is too long if it cannot be summarized in a single sentence (maybe 2)
  • criteria for too long:
    • physical length (scrolling maybe)
    • time duration (more than 50 minutes)
    • concept amount (more than 1 focused idea)

• factors that influence length:
  • desire to include related content
  • attempting to speak to different audiences
  • attempting to defend against potential criticism or fill gaps instead of focusing on the learning objectives
  • technical or modality constraints of the lesson platform or delivery
• prioritize
  • does this support instruction of the learning objective
    • may need further feedback on appropriateness or splitting of LO across lessons
  • does this make me the author / instructor feel smart (but neutral or negative impacts on LO, or could be replaced with something else that better supports LO)

• Kaija:
  • Lesson too long: A lesson is too long if learners stop being able to follow it, if it’s too much cognitive load, etc. This could happen if there are too many individual topics, or if there’s too much detail about any one topic.
  • Factors influencing: The time in which to teach it/teaching environment. Learners’ attention spans, willingness to attend.
  • Prioritize: Think back to the overarching lesson-level learning objectives and make sure they’re being met. Try out the lesson and ask learners which portions were most or least helpful. Both formal and informal version control–keeping a “question bank” or “exercise bank” so we’re not too scared to remove exercises (can always go back and retrieve them later!)
• Karen:
  • Length of a lesson can be constrained by time to teach in a classroom, word count for written text, etc. No one section should take more than an hour to get through in a classroom, or probably about 20 minutes if working independently? I aim for minimal text, but could use a rule here.
  • Factors that influence appropriate length are difficulty/complexity and value/relevance. Difficult things in which value is not immediately apparent have to be a LOT shorter. Retaining attention is hard.
  • Prioritization should be very centered around learning objectives. If those are well constructed it should be clear what’s optional. If too much has to be included in order to reach those objectives, the objectives need to be re-scoped to match the audience better.
• Kevin:
  • Too Long:
    • User struggles to remember all of the lesson’s contents.
    • Users finds themselves looking at the time.
  • Factors:
    • Cognitive load.
    • Active versus passive content.
    • Pace of the content.
  • How to prioritize:
    • Which aspects of the material are least intuitive?
    • Which aspects of the material are more technical?
    • Which aspects of the material are required for later content?
• Madison:
  • If you think about having some sort of break every 1 hour, each episode should be able to be taught within an hour.
  • Carpentries lessons are meant to be short term committments from learners. If you can’t fit all of the content into about 4-8 hours, it is probably too long. Especially, because you often need to teach multiple lessons within a workshop. We tend to run workshops that are 12-16 hours in total (spread out over multiple days, sometimes one, 4-hour session per week for 3-4 weeks).
  • Factors influencing the length of a lesson:
    • how much content can someone reasonably fit into their brain in the timeframe alloted?
    • how much time to teach do you have
  • Prioritizing:
    • Is the concept really essential to meeting the lesson objectives
• Peter:
  • A lesson might be too long if too many learners feel like it is not worth continuing, they are learning too little for how much time they invest in learning the skill.
  • The main strategy to constrain a lesson is setting very strict and reasonable goals before sitting down to write the lesson. Make sure it says everything you think needs to be said and no more.
  • I would proritize skills I consider absolutely essential, or those that are needed to learn content in future lessons. Otherwise, consider getting rid. People also have a great ability to build on a good foundation and learn new content by themselves.
• Saira:
  • When might it be too long: We are aiming for a 16 hour lesson, so anything over that would be too long; in our practice teaching sessions, if we find we having to cut out exercises or breaks to get through the lesson, then it’s too long
  • Factors that constrain: The background knowledge of the audience will greatly influence how long the lesson takes; the number of exercises and their complexity; any material that is cumulative can take longer, because you don’t have room for someone to ‘sort of’ get something (for example: setting up a git profile) - in this case everyone has to 100% be successful
  • How to prioritize: Go back to your learning objectives
• Xochitl:
  • Too long when:
    • there are too many lesson objectives,
    • too many exercises,
    • the page length is really long.
    • Could come up with an average time per episode and see how long the lesson would end up being.
  • Length constraints:
    • Complexity of the material or number of exercises would contrain how much can get covered.
    • Cognitive load.
  • Priotization:
    • Choose which LOs are essential, which are useful, and which are a bonus
    • Which objectives build off of previous ones vs. which stand alone
    • survey what the learners are interested in/what they already know about
    • what will be most useful AND is easy to master vs dropping what takes a long time to master and may be only sometimes useful

Length Considerations

• What is essential to include?
• What can be left out if needed?
• Are there checkpoints where the lesson could end if needed?
• Can important concepts be moved up earlier to ensure they are covered?

Template for notes on pilot workshops <-- we will discuss pilot workshops more at the end of today

5 Ways to Handle Extraneous Overload - Renkl 2014

1. Eliminate extraneous material
2. Insert signals emphasizing the essential material
3. Eliminate redundant printed text
4. Place printed text next to corresponding parts of graphics
5. Eliminate the need to hold essential material in working memory for long periods of time

Review Your Text for Demotivations

• dismissive language - e.g. ‘simply’, ‘just’
• use of stereotypes - check learner profiles for stereotypes too
• expert awareness gaps, i.e. places where you may be assuming the learners know more than they actually do
• fluid representations, i.e. using different terms with the same meaning interchangeably
• unexplained or unnecessary jargon/terminology
• unexplained assumptions
• sudden jumps in difficulty/complexity

Review Your Text for Accessibliity

• Avoiding regional/cultural references and idioms that will not translate across borders/cultures
• Avoiding contractions i.e. don’t, can’t, won’t etc.
• Checking that all figures/images have well written alternative text, including writing altnerative text for data visualizations.
• Checking the header hierarchy - no h1 headers in the lesson body, no skipped levels
• Using descriptive link text - no “click here” or “this page”, etc.
• Checking the text and foreground contrast for images

Optional Exercise: Alternative Text for Images (5 minutes)

Which of the following is a good alt-text option for the image below?

{alt=“Line graph of increasing carbon dioxide at the Mauna Loa Observatory from 1958 to present”}

1. Graph of data
2. Graph with increasing lines
3. Line graph of increasing carbon dioxide in ppm at the Mauna Loa Observatory from 1958 to present
4. Line graph of increasing carbon dioxide in ppm at the Mauna Loa Observatory, Hawaii, United States, from 1959 to present including values from each year. Red line shows variation in each year and black line is average for each year. 1959 = 315.90 ppm, 1960 = 316.91, 1961 = 317.64 …

Data/Image provided by NOAA Global Monitoring Laboratory, Boulder, Colorado, USA

Exercise: Completing episode metadata (10 minutes)

Add key points and questions to your episode.
To check the formatting requirements, see the Introduction Episode example in your lesson or the Workbench Documentation

Keypoints should be learner-accessible (understandable by learners without understanding the jargon in the LOs). Can be written as answers to the episode questions.

https://github.com/carpentries-lab/reviews/blob/main/docs/reviewer_guide.md#reviewer-checklist

Key Points:

• The objectives and assessments provide a good outline for an episode and then the text fills in the gaps to support anyone learning or teaching from the lesson.
• It is important to review your lesson for demotivating language, cognitive load, and accessibility.
• To reduce cognitive load and ensure there is enough time for for the materials, consider which lesson objectives are not needed and remove related content and assessments.

How we Operate

Objectives:
After completing this episode, participants should be able to…

• Describe the role that feedback plays in the life cycle of a lesson.
• Connect with other members of the community.

Questions:

• What are the important milestones in the development of a new lesson?
• How can The Carpentries lesson development community help me complete my lesson?

Lesson Life Cycle

• pre-alpha: first draft of lesson is being created
• alpha: lesson is being taught by authors
• beta: lesson is ready to be taught by other instructors
• stable: lesson has been tested by other instructors and improved based on feedback. Major changes and updates are relatively infrequent.

Pilot workshops

• alpha pilot: a workshop taught by the lesson authors, often one of the first few times the lesson has been taught.
• beta pilot: a workshop taught by instructors who have not had previous (major) involvement in developing the lesson.

Carpentries Logo Usage

Questions that can be answered in a pilot workshop:

• How much time does it take to teach each section of the lesson?
• How much time is required for each exercise?
• What technical issues were encountered during the lesson?
• What questions did learners ask during the workshop?
• Which parts of the lesson were confusing for learners?
• Which exercises could be improved to provide more information to the instructors?

More guidance for organising/teaching pilot workshops

Exercise: join relevant channels (5 minutes)

Use this time to explore the options listed above
and join/subscribe to any communication channels that you find interesting.

Join The Carpentries Slack

TopicBox (email lists)

Direct link to the Incubator Developers list

Places to find learners (do not need to exactly match your target audience):

• Carpentries channels
• Local learners
• Members of your lab group

Key Points:

• Teaching a lesson for the first time is an essential intermediate step in the lesson development process.
• The Carpentries lesson developer community shares their experience on multiple communication channels.

---

Preparing to Teach

Objectives:
After completing this episode, participants should be able to…

• Summarise lesson content as a teaching plan.
• Add Setup Instructions and Instructor Notes to the lesson site.
• Create a feedback collection plan.

Questions:

• What can I do to prepare to teach my lesson for the first time?
• How should I communicate lesson setup instructions to learners?
• What information should be recorded for instructors teaching a lesson?
• How should information be collected as part of the feedback process?

A teaching plan outlines the structure of your teaching session, including details, estimated duration and materials needed to deliver it. For example, a teaching plan may contain the following:

• welcome, introductions and motivation for the lesson - to introduce yourself, aims and objectives of the lesson and how it will help learners
• setup - to check if everyone is ready to proceed with the lesson
• segments of teaching and exercises from the lesson and any other planned activities (e.g. group discussions, wrap-up and feedback)
• checkpoints - places where you have planned to stop to check-in with learners
• slides, figures and other visual aids needed to deliver teaching and other parts of the teaching plan
• resources/references/recommended reading

Exercise: Prepare a teaching plan (15 minutes)

Create a bullet point list or brief notes describing what you will say and do when teaching the episodes you have been focussing on during this training.

Add these to your group’s lesson design notes.

Setup instructions:

• show on the home page in the learner view
• template includes summary and setup - with software and dataset sections
• you can split them out by operating systems (or other features if needed)
• these instructions go in the learners/ folder in the setup.md file

Exercise: Add Setup Instructions (10 minutes)

Add setup instructions (in the learners/setup.md file) with a list of software/tools/data needed by participants to follow your lesson and links on how to obtain and install them.
Rather than producing a separate page in the lesson site, the contents of learners/setup.md will be combined with index.md to produce the Learner View of the landing page of your lesson.

Instructor notes:

• Show in instructor view
• has a section for each episode - any inline instructor notes on those pages get compiled here
• you can add to the instructor notes doc - this is espeically useful for things that are specific for specific episodes - those are in the instructors/ folder in the instructor-notes.md file

Exercise: Add Instructor Notes (5 minutes)

Add Instructor Notes (in the instructors/instructor-notes.md file or inline in a file - if you want to look up the fenced div for it) with an initial list of points that will help you and other instructors deliver the lesson.

As you think about instructor notes that you are adding to your lesson, you may not know what they should be in advance. Instructor notes for this lesson came to be from teaching and observing each other teaching and adding back those notes later. Useful to capture this when you want this to be taught by other instructors who didn’t develop it.

Feedback collection plan:
in your pilot workshop you are going to want to collect a lot of feedback

• important to have deidicated person to capture the info during the pilot (who is not teaching or helping)
• set of quesitons that capture the types of things you want to be collecing info on
• there is also a template for notes duirng pilot
  • includes timing, and questions learners asked, etc
• also recommend minute card format for getting feedback from learners at the end of each major section - can be quick feedback -the minute card template
• template for a post-workshop survey

Exercise: Homework

The final part of this training will focus on the skills needed to collaborate
effectively. Before that there will be a break,
during which we would like you to complete the following three tasks:

1. Teach one episode of your lesson (probably the one you have been working on in these two days). See the Lesson Trial Runs page for full details.

2. After your trial run has concluded (immediately after, or when you have reviewed any feedback you collected from learners), note down your answers to the following questions:

   • What worked?
   • What did not?
   • What will you do differently next time?
   • What will you change in your material you taught?
     We will refer to these notes when we reconvene for the last episode of this training.

3. Based on your experience teaching the material and the feedback you received from your learners and helpers, make a list of issues you have identified with the material you prepared, e.g.:

   • examples that did not work as expected,
   • improvements that could be made to exercises,
   • parts that learners found particularly challenging,
   • unexpected questions or misconceptions that came up during the trial run.

   We will return to these notes during the final training session, so please make sure you save them somewhere you will be able to find them again easily when the time comes.

What questions do you have? (15 minutes)

The homework from this workshop includes a trial run of one of the episodes you have been developing in your lesson, to a real audience.
After reading the information provided about the trial run task and thinking about this task, what questions do you have about how you should approach teaching that trial run? Is there anything you are unsure of? What resources might help you prepare for that experience?

For next part, you need to have taught some part of the lesson, to some audience, and collected some feedback. - in part 2 we review the feedback and reflect upon this experience

Key Points:

• Spending time on preparing your teaching and feedback collection will make you and your participants get the most out of your workshop pilot.
• Creating clear setup instructions as part of your lesson and circulating them ahead of the pilot is time well-invested and will give you more time when teaching the lesson.
• Instructor Notes are teaching tips that you should include with your lesson to help you (a few months down the line) and other instructors, who have the relevant topic knowledge but have not been involved in the lesson design and development, deliver your lesson more successfully.

---

End of Part 1 Wrap-up

Objectives:
After completing this episode, participants should be able to…

• Summarise the lesson development process introduced in this training.
• Describe the trial run task to be completed before the next part of the training
• Identify remaining tasks that should be completed before trialling lesson content.

Questions:

• What have we learned so far?
• What needs to be done before the next part of the training?

Polish a Lesson (20 minutes)

This exercise provides you with a chance to look back over
everything you have sketched out for your episode(s) and the lesson as a whole
and consider what still needs to be done before it can be taught.

You can use this time however you judge will be most beneficial
to your preparations for teaching your episode in a trial run.

If you are not sure how to start, consider the following prompts:

• Which of the exercises set so far in this training did you not have time to complete?
• What do you still need to add/work on?
• What can you remove/consider removing?
• How will the narrative and example data you have chosen for your lesson support teaching and assessment?
• What diagram or other visual aids could you create/add to supplement your text?

Discussion/report-out

• spent this time thinking about the trial run, what we might teach, and who we might ask to join as an audience. We have some colleagues who will also be working on the lesson project, who could act as audience and help with notetaking, etc. Also reflected on the scale of the task ahead.
• spent time massaging introductory episode, it will need to be ready for the trial run. thought about some think-pair-shares for early on and outlining the content to come. Started planning remaining material, and discussed potential learners for early pilots. Tricky because we need to learn some of the material to be ready to teach it.
• we will use grad students from the lab to trial a small part of the lesson. could be a useful model for future lesson development projects

One Up, One Down (10 min)

Provide one up, one down feedback on this first part of Collaborative Lesson Development Training.
Remember:

• Say only one thing, and try not to duplicate. This gets harder for those who come later!
• Trainers should try not to respond, only record responses (e.g. in the CodiMD). This is also hard, but important!

• April: the pace, felt like the whole thing gave us time to reflect and we moved on without it feeling like it was too long
• Ani: something you already started doing - more visuals for text. Both have addressed that and offered a solution by referring to a line more regularlly
• Eric: appreciated having the codimd that gets updated and having the reference - could get distracted and see what I missed
• Hao: some of the back and forth in days 2 and 3 with working on the github as a group - little bit of confusing about who should be doing work - felt like more of that shoudlbe organized - would help figure out what we were suppsed to be doing
• Kaija: appreciated having the intersperion of time to work on content and work on infrastructure skills - set up do to those things for the rest of the lesson
• Karen: continuing on Hao’s activity scaffolding - the codimd that had a template didn’t really match, could be better organized to match the conten tfrom the exercises
• Kevin: liked having an actual template that was set up and watching it develop as people added things
• Madison: very specific thing - teaching plan, as I as doing it, I was thinking, wait, i’m making the outline of my lesson, not sure if it was true or not, maybe connect the lesson development to the teaching plan that we are working on in this exercise is building our outline of the lesson we are going to create
• Peter:
• Saira: i feel like we are at the end and i have all this codimd text but a little lost on where i start and what the overall hierarchical structure - circular diagram that is much broader, something more specific that is a big hierarchical structure thing
• Xochitl: really appreciate the info lots of info, appreciated having time to practice, good skeleton to work off, feeling less lost, though still a little overwhelmed

Feedback Survey

End of Part 1 Survey

Github videos

Making a contribution with a pull request in GitHub interface shows how to suggest a change and not run into conflicts with your collaborator
Some of this video on github Pull requests shows how to review and merge a PR.