In my previous post, “My Beef With Badges,” I called on the emerging badging community to stop conflating our aspirations with our achievements, and to start sharing a more accurate picture of the challenges we all face. I also tried to make clear that I have been a part of the very problem I now aim to address. As I’ve been asked in recent weeks to present and engage our community around these ideas, I’ve often expressed my intent to “walk the talk” and share what we’ve been observing where I work, at the American Museum of Natural History, warts and all.
Well, I finally got approval.
But before we begin, a few caveats. These are just observations. This is not a research project. It is based on a small sample, so no one should read too deep into the results (including us). So, why share it? Why not wait until the research is performed? Quite simply, we can’t afford to wait. Yes, there is great research going on, such as the work being done by Dan Hickey and his team. But, during this early stage in the application of digital badges, we need to learn what we can from our current practices to help shape research questions and to give us something to chew on while we wait the solid results.
Since early 2013, the Museum has been revisiting and exploring the benefits of incorporating a digital badging system into select youth-serving after-school programs. In summer 2013, 131 badges were conferred (out of 155 reviewed requests) to 72 youth amongst a pool of 38 different badges across seven programs.
Throughout the prototype, we were particularly interested in learning more about the effect on the staff and on youths’ learning, and on the ability for badge earners to use them outside the Museum to demonstrate their new knowledge, skills and achievements.
We created a new website prototyping a new digital badging platform, BadgeOS, developed and recently launched as an open sourced tool by Learning Times, a company whose tools had been used by the Museum earlier in the year. Each of the seven programs had their own unique section of the site where youth could manage their profiles, submit evidence (text or images) to earn badges, “friend” one another and leave each other comments.
The final report offers further details on how the badges were designed and administered, and the key observations we took from the educator and youth surveys. Below, I share an abridged segment that, as I would in the past, highlights what worked well but then include, as is new for me, where the prototype offered new and unexpected challenges (aka, what didn’t work).
“The badges kept me focused on what I wanted to accomplish in my research.”
— Youth Survey Respondent
“The effort that is used to get something most people have never heard of is silly.”
— Youth Survey Respondent
In September of 2013, surveys were conducted of all youth (invited by email with response rates shifting dramatically across the programs). When badges were introduced, most had never experienced or worked with a digital badging system in an educational context. Adults could easily reference analog badges within the Boys and Girl Scouts while youth could quickly point to digital badges within video games. But, it was new to most, and interesting to many, to think about a way to certify skills developed, information acquired, and levels of participation within an informal learning context. At the same time, as the first quote above shows, concerns were raised — for the adults, about it taking too much of their time, for the youth about feeling left out if choosing not to participate, and for both, about trivializing learning.
The surveys were designed to assess what impact badging had across youths’ educational experience, how their personal view on badges changed (if at all), and what advice they might have for the Museum moving forward. It also separated them into two groups: those who requested and received badges and those who never requested badges (those who requested but did not receive badges were too small to be significant).
Overall, the youth were equally split regarding their sense of whether the badges had any impact on them and their learning. Of those who reported the badges having an effect, the effect was overwhelmingly positive. Purple below depicts no effect, while the colors above are positive and the colors below negative.
In summary, about half of all survey respondents found that badges had a slightly or very positive effect on their enjoyment of the program, their motivation to learn, their understanding of what they were actually learning, and their identity as science learners.
The question remains, however: what motivated youth to participate? For those seeking badges, the following was reported as the reasons why:
So, of the 38 who responded that they had received a badge, about half were motivated because they were instructed to, they enjoyed it, and/or they had completed the required work. A quarter responded that they were motivated because pursuing or earning badges made them feel good, they liked to collect things, and that the badging process itself was inherently motivating. The reasons less often mentioned were an interest in sharing them with others, whether for personal reasons or for self-promotion. So, in general, one way to understand this is that the badging system offered motivating value to the youth at a personal (the activity felt good) or programmatic level (it connected with their work in a positive way) but not at a peer level (it was not for reasons of competition or social capital) or world-wide level (it was not to advance their education or career within or outside the museum).
For those who did not seek badges, the following was reported as the reasons why:
Of the 21 who responded, half agreed with the statement “What was the point?” A quarter didn’t want to share them with anyone or didn’t have the time.
Note that the two less likely reasons to motivate badge earners were also two of the stronger reasons motivating youth to NOT pursue badges: sharing them with others. In other words, comparing the two, those motivated by personal or programmatic reasons valued, enjoyed and pursued badges while those motivated by peers or world-wide reasons devalued, disliked and did not pursue badges. In addition, those who chose to mention that they liked peer competition liked the badges while those who chose to mention they did not like peer competition did not like the badges.
This might suggest that to increase youth motivation, we need to explore how to enhance youths’ direct pleasure within the badging system (e.g. a better user interface, a technically smooth experience, enhanced gamification features like a leaderboard, etc.) and its direct connection with their work (e.g. privileges within the museum related to their new skills, opportunities that are “unlocked” as a result, etc.) while minimizing the youths’ lack of motivation by increasing badge value amongst their peers (e.g. strengthen the social network aspect of the badging system) and with potential academic and workplace opportunities (e.g. developing relationships with scholarships and school admission departments to ensure they understand the value of the AMNH badges). We also would need to think about the role badges play, fostering competition and how that aligns with our program goals.
Again, to read the full report, go here. This concludes my second in a series of badge-related posts. The third in the series will explore “A Case Against Standardized Badges for Informal Learning.”
Banner image credit: Barry Joseph
