Program Evaluation

Program Evaluation

This page is a working space for "802 Program Inquiry and Evaluation" and integrates elements of "800 Self-Regulated Inquiry and Learning".

Access elements of the program evaluation through the drop-down menu in the title bar.

Access elements of the program evaluation through the drop-down menu in the title bar.

Elements of the program evaluation are accessible through the drop-down menu or click the link below to jump to a page

This entry is intended to give some background to the origin of the idea for a study into EdCamps. I have attended several Edcamps and organized a couple as well. It is a very attractive form of professional engagement but what is it really, and what is it intended to do?

Traditionally weekend events, Edcamps draw people who are interested enough in teaching and learning to give up the better part of a Saturday to talk with other teachers. While there may be a theme for the Edcamp (administration, technology, art, middle years, etc.) the actual content is not determined until the morning of the event and is driven solely by the participants. This raises a couple of interesting questions: Is this professional development? Is it more appropriately called professional learning? Or maybe professional engagement?  Does it really matter what it's called or what it accomplishes?

O'Brien and Jones (2014) discuss the debates and decision making that went into naming an academic a journal that explored professional development/learning. In it they suggest that "Professional Development" (PD) is closely related to institutional requirements, directives, and initiatives. "Professional Learning" (PL) on the other hand, is more individual. Pentland (2014) differentiates "Professional Engagement" from "Professional Exploration". Engagement occurs among close colleagues and familiar topics. "Exploration", Pentland says, occurs with individuals outside our usual realm of experience; it pulls ideas in from beyond.

Whatever we call it, Edcamps have received a warm embrace by the education community. At the time of this writing (Jaunary 31, 2016) there are well over 1000 Edcamp events from countries around the world listed on the Edcamp wiki since the first in May of 2010. As a free weekend event, it draws a self-selected group of keen and engaged participants who extend invitations to share experiences and explore possibilities in their chosen field. With this in mind, it seems the stakes are low: if participants learn something that's good, if not, it's no big deal. There was no investment made, no directive from an employer, no initiatives that required attention. Participants have their own goals and it's up to each individual whether they are met or not.

Some districts and school divisions are also embracing the model offering Edcamp experiences during the school day. These events now have significant cost attached the greatest of which are the wages for all participants. If we are concerned about accountability and value for the public dollar, which we should be, there is a need to determine the benefits of these experiences. Do Edcamps have any measurable benefit to education?

To measure the effectiveness of the program, we need to understand what the goals are so we can measure if they have been achieved. Searching the phrases: "goal of edcamp", "goals of edcamp", "edcamp goals", "purpose of edcamp", "purpose of an edcamp", I gathered about 80 statements from Edcamp events, media coverage, and blogs that mention edcamp goals. Three themes emerged:

  1. Idea Exchange
  2. Professional Networking
  3. Profession Learning

Perhaps the structure itself is the goal - Edcamps can have their own goals, participants can have their own goals. However, if there are common themes related to the goals (idea sharing, professional networking, and professional growth) there are probably ways to identify those elements that contribute most to achieving that.

Can the goal be simply those three things? Is there a need/obligation to determine how those ideas manifest in classrooms? Or determine how they affect student learning?

Below is a sample of some of the notes and thoughts that were processed early in the program evaluation.

What are the key elements of the activity we will measure?

  • engagement with others at the event
  • engagement with ideas at the event
  • change in attitude about teaching (excitement, motivation)
  • growth of professional network

What assumptions hold these elements together?

  • teaching can be an isolating experience
  • there is value in teaming and professional connections
  • professional learning networks contribute to professional growth and student learning
  • social connections contribute to the spread of ideas and innovation
  • edcamps contribute to professional engagement, serves to build professional networks, and facilitate the spread of ideas which - maybe - contributes to student learning though “the more immediate goal is to improve teacher skills, knowledge, and practice.” (Haslam 2010)

Who will be interested in the study?

  • edcamp organizers - validates (or not) their implicit belief that edcamps are valuable
  • edcamp participants - assures (or not) that there is benefit to their practice
  • administrators - offers confidence (or not) in the utility of the edcamp model for PD

What resources are available for evaluation?

  • me and some release time with EdLeave. Will recruit edcamp volunteers

Who will work on the evaluation?

  • will trial with MY7Oaks, revise with EdcampWPG, then revise and release for mass distribution

 

What is professional development?

Is an EdCamp professional development? or is there a better description for what it is and what it does?

“There is a significant difference between the systematic career progression associated with professional development and the broader, more critically reflective and less performative approach to professional learning.”

[[[O’Brien, J., & Jones, K. (2014). Professional learning or professional development? Or continuing professional learning and development? Changing terminology, policy and practice. Professional Development in Education, 40(5), 683–687.http://doi.org/10.1080/19415257.2014.960688]]]

 

“professional learning is a better way to epitomise the key characteristics of reflective practice, critical evaluation and continuing learning”

 

What is the goal of an edcamp?

 

How to measure effectiveness of an edcamp?

 

table from Vuorikari, R., & Scimeca, S. (2013). Social learning analytics to study teachers’ large-scale professional networks. In T. Ley, M. Ruohonen, M. Laanpere, & A. Tatnall (Eds.), Open and Social Technologies for Networked Learning (Vol. 395, pp. 25–34). Springer Berlin Heidelberg. http://doi.org/10.1007/978-3-642-37285-8_3

[[[" a network aimed at teachers’ professional development

where building knowledge takes place in a cultural, social and technological setting"

The article also mentions the benefit of weak ties to the spread of ideas. x-ref with Pentland]]]

 

Individual goals for attending edcamp will differ one from another. In our case, the edcamp is part of the professional obligations of the participants - it is run by the district and takes place during the school day.

Participants are prepared beforehand and arrive with a goal or two (topics they wish to explore). A pre-event survey can determine what the goals were, how much they wished to meet them, how important the goals are to the individual and compare to their sense of having met those goals after the event.

 

How important are measurable outcomes from professional development? How can value or benefits of EdCamp be measured? How can accountability for expense of public funds be assured?

 

How are edcamp organizers gathering data on effectiveness? What kind of data are they gathering? What can we infer about priority goals from the kinds of questions asked?

 

BIG PICTURE: Perhaps the goal of Edcamp is to create connections and explore ideas. Build on Pentland’s characteristics of highly effective teams, Gee’s notion of affinity spaces, [[look up other thinkers and researchers in learning communities]]

Edcamp is not the answer to all individual or organizational professional development need, but perhaps it does serve a valuable function that is not met with traditional PD.

 

Four tenets of effective PD (Bransford et al, 1999, p27)

  1. learner centered (ask participants what they need)
  2. knowledge centered (go beyond mimicry to deep understanding)
  3. assessment centered (follow up change with inquiry)
  4. community centered (opportunities for continued contact)

Five critical characteristics of PD (Desimone, 2009)

  1. active learning
  2. coherence
  3. collaborative participation
  4. content focus
  5. duration

What do you think? Share you thoughts below...