Developing a Common Language to Assess Impact

By Dana Chinn, Director of the Media Impact Project, and Johanna Blakley, co-Principal Investigator of the Media Impact Project

This year BRITDOC had five winners of its Impact Award - not one winner and four finalists as in previous years.   Was this yet another example of everyone getting a trophy just for showing up?   We don't think so.   We think it was a reflection of the increasing complexity of measuring media impact, and a recognition that "impact" can't be measured by counting discrete things such as the number of attendees at screenings.      

The Impact Award recognizes documentary films that have had "significant and measurable social or environment impact."  Just by looking at this year's winners you can see that it'd be ludicrous to select one winner by some numerical formula of "most" or "best."  

• American Promise: looks into the lives of two middle class black families as they wrestle with issues of race, class, gender and opportunity in education and navigate the ups and downs of parenthood 

• Blackfish: reveals the cruel treatment of killer whales in captivity
  

• Granito:  How to Nail a Dictator: explains how an earlier documentary film helped convict a dictator of genocide in Guatemala

• No Fire Zone: presents evidence of the systematic killing of thousands of civilians during the final days of the Sri Lankan civil war

• The House I Live In: captures heart-wrenching stories at all levels of America’s drug war
  

The full impact of any media content can only be captured over time.  How much time?  The answer is always "it depends."  A breaking news story or a TV spot?  With the web and social media, at least a day and perhaps two weeks.  A documentary film?  Years.  Unlike the Academy Award, a film can be considered for the Impact Award for five years after it's released.

In October the USC Media Impact Project partnered with BRITDOC on a workshop with filmmakers and funders to start the discussion of how to categorize - not simply count - all of the various indicators of impact of a documentary film.   We wanted to develop some standards, but not so it'd be easier for BRITDOC to pick a winner.   Standardized measures - or, a common understanding of how to talk about measuring - help filmmakers and funders alike develop campaign strategies that optimize impact.

The group soon realized that first we needed a common vocabulary.  Here are some of the terms that caused confusion and misunderstanding in various contexts:

• Engagement vs. outreach vs. influence vs. impact
• Activities vs. outputs vs. outcomes
• Groups vs. communities vs. institutions
• Behaviors vs. actions
• Macro vs. meso vs. micro
  

We also recognized that categorizing types of impact indicators into lists or a chart structure was probably too static or linear.  Cara Mertes, director of the Ford Foundation’s JustFilms initiative, suggested we look at the two-dimensional matrix or grid developed by Ethan Zuckerman, the director of the MIT Center for Civic Media, and explored by Micah Sifry in "Civic Tech and Engagement: In  Search of a Common Language." 

This matrix puts indicators in a more dynamic and relational model, one in which a film can start in one area or quadrant and then be assessed on the direction it moves given its target audiences and goals.

From Micah Sifry:  "Thin kinds of engagement require little thought on the part of the doer; thick means the participant has to really consider and deliberate.  Symbolic engagement is centered on voice, while impactful changes measureable outcomes."

Here's a stab at placing this year's BRITDOC winners in the grid:

There's no good or bad with this grid, just an assessment that acknowledges, among other things, what indicators of impact can be attributed to a film given its social impact goals and what it's asking its target audience to do.  For example, Blackfish's original goal was to "create mass public understanding of a previously unrecognized issue [mistreatment of SeaWorld whales]," which perhaps implies that it started in the top left quadrant.  However, it prompted marketing partners such as Taco Bell and potential customers to boycott SeaWorld, which resulted in SeaWorld changing some of its practices.

All Impact Award winners and finalists are excellent documentary films, but not all excellent documentaries have "significant and measurable social or environmental impact."  Impact usually doesn't happen organically.  With the exception of Blackfish, the other four winning films had outreach or campaign budgets that were 50 to over 80 percent of their film's production budget.   

Clearly, maximizing the impact of a documentary film takes substantial investment and requires time and marketing expertise, which a film's producers and directors may not have.  Ideally a film would have an "impact producer."  "Just as films have producers to manage the creative and financial process from script to screen, they also need Impact Producers to take the film from completion to impact." (BRITDOC "Impact Field Guide") 

A running theme throughout the USC-BRITDOC workshop was the need to cultivate "impact literacy," regardless of whether there's a large campaign budget or a dedicated impact producer.  If everyone agrees on the goals, knowing how much impact we have now and how much further we have to go is worth the investment.