By Sheila Leddy, The Fledgling Fund
Many filmmakers, funders and other stakeholders are wrestling with the question of whether and how to assess the social impact of creative media, and especially documentary film. At Fledgling we think about and discuss this issue a lot. Let me share some of our thinking. We recognize that not all documentaries have social change goals, rather they exist as works of art to be consumed and enjoyed, not to be assessed in the context of social impact. However, for those films and filmmakers with social change as a key goal and especially those that have decided to build outreach and engagement campaigns around their films to deepen the connection that audiences have with the social issues presented, how we define and measure progress towards these goals are important pieces of the puzzle.
As a funder focused on supporting social impact strategies for documentaries, we have a special responsibility to be thoughtful about our approach to assessing impact and recognize that this increased focus on measurement has revealed valid concerns and push back from filmmakers. Overall, we believe that the proliferation of tools and platforms (and the debate they have inspired!) are healthy and important for the field. There are now more tools in our collective toolbox to document, track, and communicate the social impact of documentaries and their campaigns. The trick is how they are applied. When used thoughtfully, they can lead to insights that then can be used in real-time throughout a campaign’s evolution; they can help shape and strengthen campaigns as they unfold. This presents an important opportunity to filmmakers and their teams; they share information with key partners along the way, deepening those relationships. This can help funders and other partners understand the role their support has played in the project to date and perhaps lead to deeper support.
We encourage our grantees to create an evaluation plan, with input from partners and funders, that is clearly linked to their own distinct impact goals and strategy and relies on different kinds of data that can help track key indicators of their progress over time. While quantitative or numerical data may be easier to come by and compare, qualitative data, which is more descriptive and observational, in many cases is more appropriate to capture the complexities of social change. With this quantitative and qualitative data, filmmakers can create stories (which we know they can do!) about the impact they have had. An “impact story” allows for deep context that cannot be achieved with numerical data alone; it can be customized to specific social change goals and can share those campaign goals as well as rich information about the issues and existing social movements that provide context to the numbers.
We should note here that it is important that any evaluation methodology distinguish between “attribution” and “contribution”. The vast majority of social issue documentaries and their engagement campaigns are entering into a community (however small or large) of activists, leaders, organizations and coalitions that have laid groundwork long before the films and campaigns were conceived and they will be there for many years continuing to build the movement. This critical work must be acknowledged - just another reason an element of “story” is so important in this evaluation work - so that this relationship can be explained.
Think of the case studies that have been shared by Britdoc, Media Impact Funders, Active Voice, Harmony Institute, Fledgling and others that rely on both quantitative and qualitative information to create a more complete picture of the impact that a film achieved, sometimes over a long period of time. Often these kinds of “impact stories” offer a more complete and useful tool for understanding the power of social issue documentary.
We recognize that creating these case studies can be time-consuming and resource intensive, particularly if done retroactively. However, we believe that filmmakers who are trying to achieve social change, impact producers and importantly funders need to make a commitment to supporting data collection and impact tracking from the moment that campaign planning begins. This front-end quantitative and qualitative data collection will not only inform the campaign but also make impact reporting more efficient and accurate. We are not calling for extensive and expensive evaluation strategies – simply for thoughtful collection of diverse data throughout the life of a campaign that will show broad sample evidence of the ways that the project is affecting the world, communities or individuals, according to stated goals. We also believe that having these conversation with funders and partners on the front end helps to build a shared understanding of constitutes impact for each party involved, ensuring that everyone is on the same page from the beginning.
We should also point out here that we believe “measuring impact” is not the best description of what we are talking about, as it seems to imply a comparison between other film projects/campaigns or against arbitrary benchmarks. When we think about data collection and assessment we are seeking to understand the success a project has in meeting its stated goals. The importance of the link between a project’s goals and strategy and its impact assessment plan cannot be understated.
We are thrilled that this field is making solid progress in our collective understanding of how films and engagement campaigns make change in the world. The emerging tools increase the options for filmmakers and offer insights into how their creators define impact. Ultimately though, the utility of these tools for individual projects depends in large part on a project’s goals and its strategy for change. We will continue to encourage and support our grantees to tell the stories of the impact they are having – using a mix of data to provide deep context. We are deeply aware that the impact of this work is incredibly complex, often happens over long periods of time and can not all be captured. However, we have seen over and over again that evidence of impact is incredibly useful and continues to move this field forward.
Sheila Leddy is the Executive Director of the Fledgling Fund.
"[Documentaries] have singled out some of the most vital issues facing Americans today. To these issues they have attempted to bring information, clarification, and more importantly, a point of view…There has been a growing appreciation for the need for different standards against which to evaluate the performance of the documentaries." [emphasis mine]
As a media strategist, I’ve been talking to social change filmmakers about social change funders (and the other way around) for many years now, but very few issues make all eyebrows go up more than “metrics.” In my unscientific and subjective opinion, these expressions of consternation are getting more pronounced as a pendulum swings once again towards the indicators of change. (The topic can also bring looks of excitement, anticipation, curiosity and incredulity, but that’s another essay.)
It’s time for a more – dare I say– measured conversation among filmmakers, funders, social scientists, artists and advocates about what we mean when we talk about impact.
Social scientists – with or without participation from creatives– have been hammering out methods and metrics for assessing the contributions of non-fiction storytelling for a long time. Researcher Elmo Wilson, trained as a journalist and quoted above, wrote about “different standards” for documentaries in 1948. When I commissioned an assessment of The Heart of the Matter Community Campaign, based on the film by Gini Reticker and Amber Hollibaugh) in 1994, there was already a body of work to build on.
So what’s changed? Amid the 24/7 media deluge, stories that speak to the heart are especially prized. Captivating documentaries – in many forms– are being used today to prompt action on every social issue. New technologies have made it easier to produce, distribute, interact with and gather numbers associated with their influence. That’s one reason why neuroscientists, social entrepreneurs and policy wonks are paying attention as never before. Before we consider whether and which new tools to use, we need to think about what it is that we really want to know. And we can’t assume that the magic of a gut-wrenching, funny and/or searing film can ever be fully “counted.” With that in mind, here are a few ways to delve in.
1. What’s the Change You Want to See?
Metrics mean “a system or standard of measurement” and thus, are not for every kind of documentary campaign. (Some of my favorite films are works of pure art and, happily for me, defy any calculation.) If you’re hoping to contribute to a specific social movement or social change process, ask yourself: what change do you hope to see in, say, one or two years? Do you want to tell a story that will help people with different kinds of perspectives come together, as a Rake might gather various leaves? Or, with the precision of a sharp Trowel, will you dig into a trenchant problem, followed by a clear call to action or policy reform?
Perhaps you want to transport viewers with a strong character-rich story and only a subtle hint of an underlying social problem? These Wheelbarrows have their own qualities and the method of measurement would look entirely different from that of a Trowel or a Shovel. Although many films are hybrids, identifying the strength of your story can clarify what to measure – and even whether to measure at all. (More about the Horticulture metaphor.) And keep in mind that unexpected outcomes, surprising consequences, and creative uses of your story are just as important as your best laid plans – they’re the emotional fuel that filmmakers bring to social movements. 2. Where’s the Common Ground? Several years ago, a funder who was new to media grantmaking asked me what filmmakers thought about a particular issue. I replied that I would never attempt to speak on their behalf and that they’re a highly diverse community made up of unique motivations, visions and business models. I thought it might be useful to sketch out some general categories (flawed, yes, as you’ll see below); understanding that filmmakers might crossover at certain points in their careers or refer to completely different identities.
Funders aren’t monolithic either and the comment about filmmaker identities holds for funders as well. This might sound obvious, but keep in mind that the people who make decisions about what to fund are individuals with unique ideas, attitudes about media, experiences and often pressures.
"Social-issue filmmakers have deepened foundations’ commitment to funding nonfiction films. We opened up many more funders’ interest in financing documentary films by cultivating foundation personnel to understand how film could be useful when it aligns with the pursuit of the foundations’ mission."
Before you meet with prospective funder(s), a) find out what kind of change they are working for, and b) why and whether metrics are important to them. For example, Gates Foundation and Ford Foundation have very different definitions and expectations for “impact.” (Some funders are very public about their position–read what the Fledgling Fund published last summer.) Open up a conversation that seeks common ground -- garden metaphor intended— and alignment of aspirations.
Funders who are interested in documentaries have plenty of sources for data–but if they’re considering film, they’re also looking for the emotional connections that only vivid storytelling can bring. (Maybe they’ll see something in your story that had never occurred to you.)
Some funders have heard that the “impact” money is perceived of as “easier to get” than production funds, so they’ll want to know how serious and accountable you are. The funders I know would rather have a candid conversation about the “meaning of metrics” than hear an outpouring of impact guarantees that have little substance [relevance?] or an unrealistic set of goals. Don’t tell them what you think they want to hear. Tell them what you are aiming for as an artist and an agent of change.
3. Who’s Counting?
As noted above, some funders have evaluation protocols in place that are a condition of accepting a grant. Understanding it in full will help you make good decisions about next steps.
But let’s say that you and your funder—and possibly the advocates or grantees who are eager to work with your film—are developing a way to understand how the project is making a difference. Even if you feel comfortable (or excited, or realistic) about being part of this assessment, it doesn’t mean that you’re the best person to gather and analyze the data. To paraphrase what I heard in a focus group in New York,
“So, in addition to financing, producing, getting releases and permits, apologizing to my family, finding a subject that’s gone missing, correcting the color and negotiating my distribution deals…now I’m a social scientist too?” This filmmaker would be off making her next film by the time any “change” could be detected. But in Los Angeles, another filmmaker said: “It’s my project and I’m the only one who could possibly determine what counts as success. That’s my next step.”
We’re developing a tool to help filmmakers weigh their involvement in assessment. (Sign up and we’ll send you our “Perspectrum” once our creative advisors have weighed in.)
I’ve seen how powerful documentaries can fuel social change and I want others to understand this power better and be able to harness it for good. We have the tools. And now we need to apply them in ways that enhance our strategies, inspire new allies, engage the creative community, and strengthen the ecosystem striving for meaningful change. Instead of raising our eyebrows, let’s raise these complex, sometimes awkward issues. Let’s dig in.
Ellen Schneider directs Active Voice Lab, which builds tools — such as ThePrenups.org and HowDoWeKnow.net — that help people use stories, art, and culture to advance measurable social change. She consults on strategies for media-savvy pioneers like Atlantic Philanthropies, MacArthur Foundation, Ford Foundation, and Participant Media. Before founding Active Voice she was executive producer of the PBS series, POV.
By Elizabeth Radshaw, Hot Docs
At Hot Docs, we believe documentary film impact campaigns are just like the films themselves. They come in all shapes and sizes and for all different audiences. It would be unfair to prescribe a blanket set of measurements to all impact campaigns with documentaries. However, there are key elements that filmmakers can keep in mind when they participate on an impact campaign associated with their film that will help support future funding for future projects.
The most important thing a filmmaker can do to ensure that they will receive funding in the future on their next project is to thoroughly complete the impact project they have at hand. Our best advice is to work with an impact producer. Documentary filmmakers can certainly embark on impact campaigns on their own. However, it may be difficult to sustain the commitment over the course of what may be a long campaign. Filmmakers executing campaigns on their own may find themselves in difficult decisions to continue to run the campaign or to make their next film. There are is a very particular set of skills, networks and measurements that impact producers have honed for this work that may not be in the pocket of a filmmaker. There are many professional impact producing companies that filmmakers can work intimately with to plan and facilitate campaigns. Just has a filmmaker would hire in a distributor, and publicists to take the specialized roles of distribution and publicity, a filmmaker is advised to hire on an impact producer to take on the very specialized role of impact producing.
The golden rule of goal setting has always been SMART: Specific, Measurable, Attainable, Relevant, and Time bound. The same should be said for impact campaigns. When working with your impact producer, the ambition and passion to bring upon change must fit within this simple guiding principle and of course with in the mission of the story with in the documentary.
Be sure to differentiate the impact goals from outreach and marketing initiatives. These sets of goals are intertwined but different. Clarity in these goals and the actions you take with your team will make reporting easier and understandable for funders.
When communicating to funders the film impact plan, make space for risk and even for failure. Funders are certainly concerned with results and proving the results through measurement, however, reach goals and some cautiously optimistic risk is very healthy for any strategy plan and certainly an impact campaign. There will be failures and it is important to detail in your reports and learn from them.
Accurate data and meticulous reporting attracts money in any industry. The proof and results of an exercised plan create a sense of trust and establish proven track record for you and your team. Detailed outcomes (successful and some failures) illustrates that you and your impact producing team can see a plan through to the end.
These few key elements to keep in mind over the course of creating a bespoke impact strategy with your impact producer will lead to successful results and future funding.
Elizabeth Radshaw is Industry Programs Director for Hot Docs.
By Michèle Stephenson and Jessica Jones, Rada Film Group
Pushing the art of nonfiction storytelling is at front and center of our work as filmmakers. By being true to that craft and faithful to its ability to expose the core of our common humanity, we believe that deeper authentic community dialogue around particular issues can be achieved. Visual storytelling has the power to evoke reflection and even individual behavior change. Combined with a carefully designed engagement campaign, documentaries can create significant societal impact. A successful engagement strategy includes trusted partnerships, strategic messaging, and thoughtful engagement both on and offline - all to deepen the relationship with the target audience. Metrics can help us understand and convey the extent to which we can create this kind of impact with our target audience. What we have learned with our latest feature documentary, American Promise, is that the measurement of impact must itself be thoughtful, and cannot be limited by only quantitative data, as it does not always represent the most meaningful kinds of engagement. Instead metrics should be thought of as contributing to an overarching analytical narrative that examines one on one dialogue sparked by a film’s issue(s), and an analysis that sheds light on the better uses of storytelling that help push for both individual behavior and systemic change.
"Developing strategic partnerships is essential for both funding and successful engagement. With American Promise, we saw an opportunity to contribute to the efforts of organizations already building a movement to close the black male achievement gap in education." Strategically designed focus group research and discussions supported by entities such as The Fledgling Fund and the Ford Foundation during the production of American Promise, resulted in trusting partnerships with key organizations that remain instrumental to the success of the film’s campaign, continuing to this day. By the time American Promise’s release occurred, a total of 66 national partners and 118 community organizations had mobilized to support the film. Stronger partnerships between organizations were also forged as a result of their collaborative use of the film; for example, United Way and America’s Promise Alliance seized the opportunity to work with The Arthur M. Blank Foundation to deepen their work in Atlanta by using American Promise, and supporting our participation in a two-day long public workshop series. These types of collaborative opportunities sparked by the resonance of the film’s story and community engagement effort sparked local funding support in a number other cities: Omaha and Minneapolis to name a few..
The initial partner conversations we had in the planning phase also helped narrow our priority target constituencies – African American parents and teachers of black boys between birth and age twelve. With specific target audiences identified we focused on understanding their needs, and worked to develop specific tools for engagement, including a book, a mobile app, screening guides, and workshops. We embraced an evidence-based approach to develop the impact metrics of the campaign, such as after-screening surveys, partners’ metrics, and online data analysis (webviews, google trends, etc.). For example, the term ‘implicit bias’, a main point of discussion in our speaking engagements and workshops, was used on average 4 times daily in news headlines prior to the campaign, and is now used 20 times on average and continues to rise. Though we can not attribute this increase entirely to the American Promise campaign, we believe we have contributed to the public discourse and increased awareness around this issue. In conjunction with these metrics, we seek qualitative measurements such as interviews, quotes, personal narratives and empirical evidence gathered in person and online. Not only does this allow us to contextualize some of the data collected, it is a robust demonstration of our target audience’s personal and often deep commitment to change.
Examining the reach of our target audience is not only critical to quantitative metrics, but also valuable to our understanding of a much harder to measure metric – gauging a film’s ability to spark conversation that affects individual behavior and attitudes. This understanding was developed over time with offline community work aimed at building trust and engaging in direct community dialogue that can at times be uncomfortable but necessary. These conversations gave us a first hand idea of individual and community needs, and served to expand and deepen our campaign tools, such as our Promise Clubs, a national parent affinity group network, which directly benefits our target community. We continue to learn about parent groups, small community screenings, group meetings, and schools not directly related to our outreach work that have incorporated tools associated with the American Promise campaign. We can only assume many more have occurred across the country that we are unaware of. Paradoxically, a main goal of our community engagement work is precisely to push for our target communities to take ownership of the campaign tools and instigate their own smaller level engagement work. Yet, once these interactions are sparked, we in turn relinquish a certain level of control and ability to measure their impact. This direct offline community interaction combined with the personal vulnerable nature of our story continue to have a hard to measure exponential ripple effect that is deep and long-lasting. Just recently in our third trip to Atlanta, the rich conversation that occurred after a public screening at the Atlanta International School extended to well over two hours. If not for the plane, we would have stayed longer. Smaller parent and educator groups formed to continue the exchange throughout the coming months and next school year. This direct engagement highlights the potentially unquantifiable emotional impact and continued conversation a film can trigger, and is a better demonstration of impact than easily digestible numbers such as attendance, or demographics. These exchanges speak to the heart of ‘taking action’ and inform our impact metrics narrative.
The most meaningful moments of impact are often captured in conversations with our target audience and feedback from our partners. “[We wanted] to dig deeper in our work, but we needed a narrative, a human story that could serve as a catalyst to make our work happen in an inclusive way that does not point fingers” said a school district representative in Ft. Lauderdale, Florida. In many cases, American Promise has provided a way to reach that pivotal point of discussion where a community can lean into a difficult conversation on the road towards change. It is crucial to demonstrate to funders an understanding of the audience, and to discuss the measures for campaign success based on the audience’s needs. When we discuss funding, metrics, and impact, we should all be asking (and many of us are), what is the intrinsic value that this analysis brings and how do we develop more comprehensive metrics that are inclusive of the entire engagement conversation.
Michèle Stephenson is the co-founder of Rada Film Group and Jessica Jones is the Impact Producer at Rada Film Group.
By Joanna Raczkiewicz, Harmony Institute
Over the last twenty years, the number of documentary films produced annually has grown exponentially. According to Numbers.com, a site that tracks the US box office revenue and movie sales, documentary films account for 14 percent of all films released between 1995 and 2015, placing it as the third most widely produced genre. Those numbers don’t account for the fact that docs are commonly distributed via broadcast television, cable networks and, more recently, online meaning the actual number films released is likely much higher.
Despite this impressive rise in number of documentaries produced, few documentary producers and fans will be surprised to learn that volume doesn’t correlate with earnings. In the same twenty-year period that documentaries captured 14 percent of the market, they made up less than one percent of market share. It’s no wonder then that investors eager to profit from financing films steer clear of the genre.
Documentarians have thankfully learned to look elsewhere for funding. Many philanthropists and mission-driven organizations working for social change see the value of documentary projects as going beyond strictly financial. Whether communicating fundamental truths of human experience, or putting complex issues into personal context, these funders have realized that documentaries can engage and move audiences as few other forms can. (Empirical studies back this up, suggesting that narratives like those experienced by documentary audiences have a “privileged status” in human cognition, information processing, and recall.)
According to a 2013 Foundation Center report charting US grantmaking activity from 2009 – 2011, foundations are increasingly supporting media-related work across multiple fields, from citizen journalism to communications infrastructure. Accompanying this trend is a growing emphasis on accountability and data-driven measurement among philanthropic organizations, with terms like “impact investing” and “outcome-oriented” philanthropy gaining currency over every award cycle. This push towards metrics to measure program effectiveness is passed on to grantees—documentarians among them.
The problem is, grantsmanship and impact measurement aren’t skills usually associated with filmmakers and creatives. The imperative to articulate achievable goals; identify appropriate metrics; collect and analyze data; and report on outcomes is daunting to many filmmakers, many of whom cite a lack of time, resources, and training needed to do the job. Others oppose the idea that creativity can be quantified, regardless of whether they believe their project has the potential to change minds and spark social change. Setting baselines, distinguishing between causal and correlative measures, tying indicators to theories of change—this is the work of social scientists, not auteurs, after all.
At the Harmony Institute, we devote our time to studying the mechanisms through which narrative media contribute to positive social change. We conduct basic research to test social science theories of influence, develop computational methods to mine online discourse and behavior, create evidence-based tools for practitioners, and advise on strategy and evaluation.
Our work in the field has taught us the value of starting the impact-planning process early. It also inspired the creation of StoryPilot (currently in beta), an online platform designed to empower storytellers to embrace data and metrics to advance their art.
There are number of initial steps you, documentarians, can take to effectively incorporate metrics into planning and fundraising for your project. Addressing the questions listed below will go a long way toward helping you set concrete, measurable goals.
● What inspired you to develop or produce the film? Was it a particular experience or relationship? Are you building on prior work? Are you addressing an issue or social problem that you feel hasn’t received the attention it deserves?
● Who are you making the film for? What audiences do you want to reach? Why? What do you want them to take away? How do you want them to feel after watching the film? What, if any, actions do you want them to take?
● How is your project unique? Are you presenting a new perspective? Telling a story in a new or unconventional way? If you’re focusing on a specific social issue, take some time to explore how others have treated similar subjects. (Check out StoryPilot.org to explore the landscape of recent social-issue documentaries.)
● Why are you the best person to tell the story? Do you have specialized knowledge of the subject or access to data and resources that other may not?
The next step is to translate your answers into measurable goals in order to identify appropriate metrics—i.e., the data, information, and tangible evidence you’ll collect to gauge your progress toward their achievement. Keep in mind that the metrics are most effective when chosen and applied on a case-by-case basis, custom-fit to a particular undertaking.
At the Harmony Institute, we often hear media makers saying they want to “change the conversation” or “raise awareness” around a particular issue or injustice. These are valid ambitions, but not yet measurable goals. As detailed in the Impact Playbook, an impact manual for filmmakers and change-agents, a key activity in any measurement project is to establish baselines, or reference points against which you can measure change. For example, as detailed in a study of the film Gasland (Vasi, Walker, Johnson, Tan; American Journal of Sociology, forthcoming,) we were able to observe and quantify shifts in public discourse around the issue of hydraulic fracturing (“fracking”) by comparing and analyzing mainstream and social media before and after the film’s release and award nominations.
Regardless of whether you’ve set your sights on creating a transportive experience for your viewers; changing their comprehension of, attitudes towards, or behaviors around an issue; or achieving broader systemic change on a policy level, instruments for collecting and analyzing outcome data abound, each with its strengths and weaknesses.
If this sounds overwhelming: don’t despair. There’s a vibrant, and growing community of individuals and organizations serving the field, and myriad excellent, free resources to help guide your way (a handful of which are listed below).
Together with our colleagues in the field, we’re committed to maximizing media impact—which brings me to a final thought on garnering support for your next project. Approach potential funding opportunities as more than a means to an end. The vision and values that drive philanthropic investments are realized through their grantees; thoughtful application of metrics will help ensure mutually beneficial outcomes.
The following offer a few points of entry to the many resources in the field:
Guides: The Impact Playbook, The Impact Field Guide and Toolkit
Tools and resources: Media Impact Funders (AIM), Sparkwise, StoryPilot, TRASI
Harmony Institute: @HInstitute, The Ripple Effect blog
Joanna Raczkiewicz is Manager of Research, Strategy and Development for the Harmony Institute.