by John Fraser, PhD
Dr. Fraser is the President & CEO of New Knowledge Organization Ltd, and a Fellow of the Media Impact Project at USC Annenberg Norman Lear Center. His series of guest blogs on social media use among millennials will appear each Monday for the next three weeks.
Op-ed pieces about the cost of social media to civil society tend to be reinforced by an age-old trope that young people (no matter what generation) are enamored with their new technologies but have increasingly lower attention spans never before seen in human history. Rock and roll was supposed to lead to the demise of society in the 1950s, almost the same way that fast cars and sexualized clothing of the 1920’s were assumed to be evidence of the end of western European civilization. It seems that social media is the new rock and roll.
As new social media emerged, academics perhaps not raised with these technologies invested in a rapid-fire process of pathologizing digital behaviors. They often characterized failure as an inability of users to succeed in traditional learning modalities. It was those modalities – single-task execution or concentration in a classroom – where academics were rewarded in their own careers. They studied game addiction, comparative loss of single-task attention skills, and how social media negatively impacts academic performance.
While some studies were hopeful about the value of digital tools in society, the negative public reports far outweighed the positive. In the face of that work, our national study of teen gamers, It’s not who I am, it’s what I do broke new ground. We found that most so-called high school-age “gamers” were not all-consumed by their media tools. In general, youth had set aside a reasonable amount of time and energy for digital experience as part of their identity development process. What was more revelatory in that study was that teens were choosing their digital gaming to reinforce their ability to achieve success. Those who were more successful in science-based academic pursuits were more likely to use their digital gaming in isolation to advance their mastery, while those youth who were less academically successful with science were more likely to collaborate as a way to be successful. It is only in the past few years that we’ve seen an abundance of affirmative studies that consider how social media is changing the way we think – not better, not worse, just different.
We’ve labored too long on the assumption contemporary youth have low attention spans, when it’s now becoming apparent that most youth are simply exercising good judgment on how to use their time effectively.
Electronic Media Use, Reading, and Academic Distractibility in College Youth, Laura E. Levine, Bradley M. Waite, and Laura L. Bowman. CyberPsychology & Behavior. August 2007, 10(4): 560-566.https://doi.org/10.1089/cpb.2007.9990
The Associations Between Social-Media Use and Academic Performance Among Undergraduate Students in Biology. Jonna M. Leyrer-Jackson, Ashley K. Wilson, Journal of Biological Education. Mar 2017, Vol. 25: 1-10
Smartphones and Cognition: A Review of Research Exploring the Links between Mobile Technology Habits and Cognitive Functioning Henry H. Wilmer, Lauren E. Sherman, Jason M. Chein Frontiers in Psychology. Apr 2017, Vol. 8Crossref
Sleeping with Technology: Cognitive, affective, and technology usage predictors of sleep problems among college students, Larry Rosen, Louis M. Carrier, Aimee Miller, Jeffrey Rokkum, Abraham Ruiz, Sleep Health. Mar 2016, Vol. 2, No. 1: 49-56
“All Over the Place”: A case study of classroom multitasking and attentional performance. Dan Hassoun, New Media & Society. Nov 2015, Vol. 17, No. 10: 1680-1695
It’s Not Who I Am, it’s What I Do: Youth gamer identity and science understanding Asbell-Clarke, J., Fraser, J., Gupta, R. & Rowe, E. (2013), Transactions of the Digital Games Research Association, 1
MIP Editor’s Note: Journalism can and does spur real change. The following two stories by our MIP guest blogger outlines how two reporters spotlighted serious problems in government agencies and helped initiate improvements. The first is a heartbreaking story involving the overwrought Child Protective Services in Rochester, New York, and looks at coverage that persevered for almost two years. The second piece hails from the TCPalm in Stuart, Florida following Hurricane Irma. It looks at how one reporter’s coverage of inefficient bureaucracies inspired an overnight environmental clean-up. One long-term, one short, but both stories proving the power of local journalism to change lives.
Institutional Change in Rochester
Reporter Meaghan McDermott was covering the general assignment beat one day in November 2016 when she got a tip that police were looking into the death of a little girl at a local hospital. McDermott is a watchdog reporter who typically covers taxes and property for the Democrat & Chronicle in Rochester, N.Y., but like many reporters these days sometimes she fills in on breaking news.
For the next year and a half, McDermott stayed on the story as investigators declared 3-year-old Brook Stagles’ death a homicide, her little body bruised and broken inside and out; as Brooke’s grandfather pointed a finger at Child Protective Services; and then as the child’s father and his girlfriend, Erica Bell, were charged. She sat through Bell’s entire trial. She investigated CPS. Within days of Bell’s criminal conviction in October 2017 — almost a year after McDermott picked up that general assignment shift — Monroe County Executive Cheryl Dinolfo announced sweeping changes to the county's Child Protective Services agency.
Institutional impact such as this — a change in how an entire agency or organization operates — commonly takes longer to occur than individual action. It often means lengthy and costly reporting (staff time, public records fees, etc.), deliberation, agreement and adoption or legislation, all of which are more involved than single hirings, firings or resignations. Policy change, too, takes time. McDermott’s story and the one I’ll tell next sit on opposite ends of the timeline, yet both led to important change in their local communities. McDermott credits the impact of this story to two primary factors:
“Brook’s grandfather blamed CPS early,” McDermott said. “He started putting up billboards, websites, trying to get candidate (Donald) Trump’s attention.” He spent nearly $100,000 on billboards in cities across the U.S. carrying messages such as "Don't let children die from child abuse … like Brook Stagles" and “Child abuse crisis: Help CPS save lives. How many have to suffer or die? Brook did both!”
“His story from the beginning was much more focused on failings with CPS,” McDermott said. “My reporting later determined it probably didn’t have a direct impact on her death. Erica Bell (did). But it did make me interested in whether the system working as it should, and my reporting found it wasn’t.” McDermott collected data on CPS cases and deaths, including documents showing caseloads in Monroe County were some of the largest in New York.
“Usually you’d want caseworkers to have 12-15 cases a month, but some workers here had in excess of 50 a month,” McDermott found. Monroe was 54th worst of 60 counties in overdue investigations, 40th worst in number of workers with more than 15 investigations.
Yet as it does in the best investigative reporting, the human drama proved more powerful than numbers. For McDermott, that was her coverage of Bell’s trial. She was there reporting every day. The local TV station covered the trial, too, but it was McDermott who did the digging. (Immediately after the trial ended she published “Little Girl Lost: The questions left behind by Brook Stagles' death,” a look at the “litany of failures and missed opportunities” in Brook’s case.) “I’d like to think that the coverage generated enough outrage that it moved the needle,” she said. “Over time, the horrific detail of this girl’s death and what happened was so shocking to the community that I think the county felt they had to do something.”
In all, the county:
A mess in Hurricane Irma’s wake
About a month before Bell’s conviction up in Rochester, Hurricane Irma barreled through south Florida and rattled the windows in the TCPalm newsroom in Stuart, Fla. By Sept. 18 it was gone, and Miguel Gonzales Sr. was on the phone to report a bunch of dead fish in the canal behind his house. Tyler Treadway, a reporter on the newsroom’s Indian River Lagoon team, had been on the lookout for fish kills and other storm damage. He asked for proof, and Gonzales sent photographs. Treadway’s story explained what had happened after Irma — the science of it:
“Rain from the storm flushes large amounts of nutrients into the water, causing bacteria to have a feeding frenzy that sucks oxygen out of the water. Low oxygen levels cause the fish to suffocate. Oxygen levels in the C-24 ‘took a nosedive following the hurricane,’ said Edith ‘Edie’ Widder, founder and lead scientist at the Ocean Research & Conservation Association in Fort Pierce. ‘I'm not at all surprised there was a fish die-off in the canal.’”
The smell was awful, but that was only part of the problem. As fish die they release nutrients into the water that can feed algae blooms. Photographer Leah Voss took pictures of the thousands of dead carp, bass and striped perch, a sea turtle and a six-foot alligator stinking and rotting in the sun on the banks of the canal. The next day, Treadway started asking who would clean up the mess and a “reverse turf war,” as he calls it, broke out. Three different agencies — South Florida Water District, Florida Fish and Wildlife Conservation Commission and the City of Port St. Lucie — denied responsibility.
So he wrote about that next. Publishing the second story “kind of shamed them,” Treadway said. Impact was swift. The next day, the Water District agreed to take the lead and all three agencies came together to get rid of the fish. Just two days after Gonzales called to complain, the fish were gone.
“What I love about this story is it’s not the biggest story we’ve ever run but it’s what we do,” said Treadway’s editor, Adam Neal. “We got a call from a reader, and then we took it a step further to ask who’s going to clean this up. If Tyler hadn’t made those phone calls, I can almost guarantee it never would’ve happened.” Eve Samples, TCPalm’s opinion editor, also credits Voss’ photographs — pictures of piles of rotting fish — with inspiring impact.
“The photos were so horrifying they really helped tell this story,” Samples said. “I think because of those photos it was very widely shared on social (media) -- because of the yuck factor.”
Read more about the local impact TCPalm has had in its “12 Days of Christmas” campaign benefiting nonprofits who help the lagoon.
Anjanette Delgado is the senior news director for digital at the Detroit Free Press and freep.com, part of the USA Today Network. The Democrat & Chronicle and TCPalm are also part of the USA Today Network. Email: email@example.com, Twitter: @anjdelgado.
by Laurie Trotta Valenti
A year ago, MIP Senior Fellow Jessica Clark unveiled a hands-on tool for helping media makers and funders map outcomes of a project: the Media Engagement Strategy Deck.
The cards work along the lines of a tarot. Different suits help users puzzle through key concepts for engagement, such as the who, what, why and how of a project, represented by symbols and colors. Connector cards allow users to construct a layout almost like a math equation: “You can begin with the factors that you already know (either your desired outcome or your starting platform) and then build an impact story from left to right, or right to left,” Clark says. “The language around media impact planning can be intimidating,” she explains. “The cards are designed to be friendly and intuitive, a way to plan and evaluate an engagement campaign.”
Clark designed the cards to help producers and funders conceptualize the trajectory of an engagement campaign, and then use the decks to talk to their teams about project goals, shared outlooks, and differences between the two that might exist. The cards are meant to be flexible and reflect the many different creative approaches that media makers are taking to draw communities into issues.
MIP checked in with Clark during this anniversary season to learn about the deck's reception among her colleagues in nonfiction storytelling, academia, funding and beyond. We wanted to learn about the different contexts in which the cards are being employed and whether they are proving useful.
“We’ve had enthusiastic response from people in newsrooms and documentarians,” Clark reports. The cards are being used in different ways. One Sacramento journalist used the deck on a documentary project, A View From Here, which looks at a range of social issues over time, such as food insecurity, immigration and housing. The editor used the cards to help journalists embrace more community engagement inside the newsroom. “We were trying to move the journalists toward a mindshift,” Clark says. “We’ve been in dialogue about the theory of change for the organization, the line between advocacy and journalism, and the cards help illustrate the concepts.”
The range of places and organizations employing the Media Engagement Strategy Deck include: among media makers at AFI Docs and Double Exposure; at funder gatherings such as ComNet and Philanthropy Workshop; in classrooms around the globe, including Central European University, the University of North Texas and SUNY Purchase; by journalism outlets and conferences such as Capitol Public Radio and the Online Newsroom Association; in media venues overseas, such as the Jakarta Global Forum for Media Development; with PhillyCam, a media literacy training program, and in one-on-one sessions with funders, researchers and media makers. The cards are also being adopted more rigorously by some groups: The Solutions Journalism Network has worked them into their training curriculum, and Greenpeace International’s digital trainers have ordered several decks.
Last summer, Clark issued an updated version of the deck that includes two new cards: trust and collaboration. The “trust” card is based on “the many conversations we’ve been having around the country on lack of trust in the media,” and falls into the “outcomes” category of the deck, Clark says. The collaboration card grew out of a need to add partnerships as an option in the connectors category, she adds. One of the more unusual suits reflects the growth of experiential media, such as virtual and mixed reality. “The cards are for the senses, the experience of the user; they chart the eventualities of an engagement campaign. They are a living expression of the world.”
Want to try the cards for yourself? Purchase decks here.
by Johanna Blakley
Lately I’ve been doing a bit of VR bingeing. While I’m not a binge TV viewer, I cannot get enough VR when the opportunity presents itself. Luckily, I’m based in Los Angeles, where new media experiments are all the rage, and my job takes me to exactly the kinds of conferences that showcase new experiments in the field.As you’ve no doubt heard, VR is all the rage right now, syphoning start-up funding from what many believe are more worthy, though less glitzy, projects. Stanford’s Jeremy Bailenson has just published Experience on Demand, which catalogs the many uses for VR, mostly for training or therapy, and VR pioneer Jaron Lanier has just come out with a surprisingly optimistic book, Dawn of the New Everything. Even if you’re skeptical about VR, I strongly urge you to try it out. And not just what you can pop into your Google Cardboard. Before you completely dismiss it, you need to experience it with better hardware, such as Samsung Gear or, even better, the HTC Vive.
Unfortunately, that’s easier said than done. Even if you get the chance to go to South By Southwest or the Future of Storytelling Festival (I went to both last year), you will soon learn that you’ll be facing long, depressing lines, sometimes hours long, for an experience that will last less than ten minutes — sometimes much less. Welcome to the dark side of experiential storytelling.
However, where there’s a will, there’s a way: ask questions, inquire ahead, and see if you can make reservations. IMAX VR now has 6 locations and counting: I loved Eagle Flight and Raw Data, two exhilarating multiplayer games. And museums and cultural centers offer these experiences, as well, often with a more civilized reservation system in place than conferences offer.
And then sometimes you just get lucky. While strolling through Montreal’s old city, I saw a sign saying “Luxury Rubbish:” anyone who knows me knows I wouldn’t pass that up. Turns out it was the gift shop for a cultural arts center that just happened to be hosting a massive VR exhibition curated by none other than the Future of Storytelling.
Needless to say, I showed up the moment the place opened, on a day the staff recommended as a low-traffic day, and spent half a day in other worlds of many people’s making. Afterwards I knew I must share a guide to these experiences so that unlucky others can figure out which line they should get into at the next SXSW, Sundance, Tribeca or Future of Storytelling Festival.
Instead of offering reviews of quality and preference, my goal here is to give some indication of what new realities these VR pieces explore: are you interested in how VR might transform books, art, dance, games or movies? Then read on, my friend.
by Johanna Blakley
Washington Post reporter Krissah Thompson contacted me recently asking me if I had any thoughts about Melania Trump’s use of social media. I thought it was a fascinating question and so I started digging into her Instagram and Twitter accounts.
I must say, I found both feeds really dispiriting. I suspect that a lot of people who follow her closely and rate her highly might feel sorry for her. There is very little evidence that she is sharing anything remotely personal, which is what primarily attracts people to the social media feeds of celebrities and other powerful public figures. (The big exception, which might prove the rule, is her last Instagram post, a flirty close-up with a Santa hat.)
You might assume that Melania Trump is simply uninterested in the public stage or is uncomfortable sharing her private life with the public, but her personal Twitter feed, which she hasn’t updated since election day, is filled with personal preferences and observations that feel quite intimate. So much so, that I’m pretty sure I could pick out a gift for her (and flowers!) that I know she’d like. That is what’s so powerful about these social platforms: they can make you feel as if you really know someone. But the carefully coiffed woman featured in the FLOTUS feeds seems distant and disconnected. I couldn’t help but wonder whether her approval rating is higher than her husband’s precisely because she withholds so much, which gives her the patina of dignity.
Of course it’s tempting to compare Melania Trump’s social media presence with that of her predecessor, Michelle Obama. While the former first lady’s feed was also loaded with official events, she often spoke at those events, and through various media outlets, which allowed her to post material filled with her voice, her attitude, and her humor. She also had the habit of posting personal musings in the official FLOTUS account, which were differentiated from the rest of the feed with the initials “mo.” You don’t see any intimate asides from Melania Trump.
The vast majority of the photos she posts are documentation of formal events: she’s basically caught on camera performing her duty. What you don’t tend to see is her looking into the camera, or trying to connect with the American public, as Michelle Obama does quite convincingly in her photos and videos. Our current first lady seems to see herself as someone to be seen – which makes sense given her professional modeling career – but strikes me as a bit chilling in her new role, which is endowed with tremendous cultural and political power. The big question is, will she ever choose to wield it?
Read Thompson’s Washington Post article.
By Laurie Trotta Valenti
A case study by the USC Annenberg Norman Lear Center’s Media Impact Project illustrates how one documentary series helped re-invigorate progressive groups following the 2016 elections, and may have lessons for media makers intent on social impact. The study of the EPIX series America Divided (which is available on Amazon and HULU) illustrates that documentarians can use innovative engagement campaigns to rally the public to action long after a program’s broadcast (if they were lucky enough to be broadcast!).
America Divided producers took 10 strategic steps to engage citizens following broadcast. Some of the steps were tried and true, such as using celebrity prestige to attract attention, but others were innovative, such as editing several shorter, single-issue versions of the series that could be screened before niche groups. This helped to build coalitions around the project as well as re-energize a demoralized liberal base following the 2016 elections.
The MIP study revealed that a key benefit was this re-invigoration of groups around the root causes of social injustice: the screenings and panel discussions gave activists a reason to assemble, to air their views and, more importantly, the paths to solutions. Specific actions that grew from these assemblages include canvassing, starting Social Media pages, creating petitions, signing-up for planned marches and rallies, as well as organizing additional meetings and screenings. The Case Study extracts 15 insights for media makers to create engagement around their work:
15 Tips for Media Makers to Keep Social Issues in the Public Eye
1. Think of broadcast as just the first step in your engagement campaign.
2. Forge partnerships with groups who can screen and promote your work.
3. Seek connections between your issue and any arts, civic, philanthropy or cultural groups and their own activist work.
4. Focus screening campaigns around one rallying issue.
5. Partner with regional PR firms that know the local players and issues and can unite disparate groups.
6. Be open to re-purposing edits that allow for issue or region specific coalitions.
7. Exploit celebrity power to cross-promote online, draw live audiences and create a relevant “face” for the issue.
8. Use screenings as forums for people to assemble, discuss and create community.
9. Engage audiences in panel discussions following screenings to promote deeper dialogue and create plans for social action.
10. Support learning with fact-based resources for deeper study.
11. Keep the momentum going by continuing to supply content to your partner groups.
12. Reach out to different-minded audiences to promote civil discourse.
13. Make the information in your documentary searchable online
14. Make certain screening partners can easily access your materials.
15. Keep your website current so that it becomes a source for news on your issues, and maintain ongoing dialogue with your digital audience.
Here is a full copy of the report; please contact us at if you would like to be added to our mailing list.
by Anjanette Delgado,
Guest Blogger for MIP
Journalists don't as a rule, have a specific impact in mind when we begin our journalism. We list project goals, but other than bringing awareness to an issue or event we do not identify what we’d like to see happen next.
This is a story about the one time we did.
Rockland County, just northwest of Manhattan, is one of New York’s fastest growing counties. Neighborhoods that used to feel suburban and bucolic now choke with high-density sprawl, multi-family homes rise in backyards next door, traffic is a problem and religious schools pop up on residential streets.
“A generation ago, there were few problems between Ramapo's small ultra-religious Jewish communities and the gentiles and other Jews who made up the bulk of the town's population. Things have changed. As the ultra-religious community has grown, Ramapo has become a flashpoint in a continuing conflict over what it means to live in the suburbs. … The conditions fueling that conflict are now threatening to spread beyond Ramapo's borders. Surrounding communities have taken notice, and they are adopting measures aimed at heading off the strife that has become the norm for their municipal neighbor. No place, it seems, wants to become ‘the next Ramapo.’”
People in Rockland County, one of two counties we cover at The Journal News and lohud.com, argue over this. Things get ugly. Reader comments with the stories we write turn hostile and border on anti-Semitism. No matter what the topic, the conversation inevitably devolves into “us” vs. “them.”
But here’s the real problem, and it’s not religion: Ramapo’s loose zoning, lax enforcement of fire and building codes and largely unchecked development puts everyone at risk.
We wanted to know: Do people understand what’s really happening in Ramapo? If we were to explain the issue — lay it out in longform story, year by year and decision by decision — could we all (or most of us) finally agree? Could we change minds? If we could achieve consensus, would we stop arguing religion and start attacking the real problems?
So, after years of having the same tired, angry, circular discussion, we set out — in one project — to change the conversation.
We researched the story over several months. We talked to longtime residents, members of the Jewish community, government officials and the county’s Fire and Emergency Services coordinator. We wrote and rewrote, shot video and mapped the region.
“Ramapo nears breaking point” is a longform piece of explanatory journalism that laid out all of the problems that began decades ago.
At the end of our story, we included a short survey to measure the impact of our
explanatory journalism — to find out if we successfully proved that the issues in Ramapo are a lack of zoning and safety code enforcement, not religion. (We considered asking for opinion at the beginning of the story to gauge change at the end, but felt that would inhibit readership and we could accomplish what we needed with one survey at the end.)
The survey was four questions, beginning with two easy Likert Scale queries:
Then we expected a steep drop-off in answers to the final two questions because the first takes time and the second raises privacy concerns:
We built the survey using PollDaddy because it prevents repeat responses and for the data it provides on the back end, including geolocation, a time/date stamp and referrer, and it assigns each respondent a unique PollDaddy ID. This helped us connect their answers from question 1 to question 2, and so on, and get a more complete picture of their response.
We embedded the survey at the bottom of the longform story. For two days, that was the only place you could find the survey; we didn’t embed it elsewhere or share it on social media. After two days we added it to our editorial, “Ramapo’s shoddy governance is by design.”
Surprisingly, nearly 900 people responded. Here’s what they said:
Do people understand what’s really happening in Ramapo? Yes. Most (67%) either agreed or strongly agreed with our explanation of the issues, with “strongly agree” ranking the highest (54%).
Could we change minds? Yes. Of those 67%, 14% said our reporting convinced them, while 35% said they agreed beforehand.
Finally, we believed we and the community could stop having the same conversation and instead move forward toward solutions.
Emails. Almost a third — 29% — gave us their email address for follow-up.
Sources and commentary. Even more surprising than the total number of respondents was the percentage of them — 38% — who also took the time to write about how Ramapo’s issues have impacted them. Hearing from homeowners, those who’ve moved away, educators, fire safety officials, people who have tried to move in but claim to have been denied housing gave us a list of future story ideas and sources.
A few wrote to thank us for our reporting, for exposing the issues and doing so in such an
unbiased way. A couple said we should have written this years ago. One called it “conversation-framing analysis” and others called for more discussion on the topic. One response shows how we raised awareness and encouraged civic involvement:
“I live in a neighboring town and am concerned about Ramapo’s issues becoming issues here, too. So I will be paying more attention and doing what I can to ensure that we do not fall victim to the same lack of zoning and code enforcement that has plagued Ramapo.”
What we learned
Anjanette Delgado is the digital director and head of audience for lohud.com and poughkeepsiejournal.com, part of the USA Today Network. Email: firstname.lastname@example.org, Twitter: @anjdelgado. Special thanks to Lindsay Green-Barber at the Center for Investigative Reporting, who helped with survey methodology.
Erica Watson-Currie, PhD: November 2017
At the Media Impact Project, our mission is to understand the effects of media on viewers. We also strive to apply acquired knowledge to projects that serve the social good, and to be a thought leader on research issues in our field. This means we assume two distinctly different roles: the “critical friend” of the evaluator and the impartiality of the researcher.
By evaluating the work of individual documentarians, journalists, and other media makers, we help find ways to improve engagement opportunities with audiences on a micro level. In studying the general effects of media on society, we contribute to the greater body of research on this topic that touches us all. These are important distinctions to make, as evaluations of specific programs or films provides us with information about the whats -- audience demographics and how audiences responded, what actions they took, and if their behaviors changed; research on that same show can help us to understand the whys of media impact -- what was it about the program that sparked the response, spurred the action, or shifted behaviors.
Myriad scholars have weighed in on the distinctions between evaluation and research. Michael Scriven's oft-referenced explanation of differences between the two disciplines posits that although both practices apply social science tools to conduct empirical investigations and analyze data, evaluators do so to assess value of what is being examined with an eye to whether predetermined standards are met. This, while researchers collect data to test hypotheses and reach conclusions based on "factual results." Into this fray we contribute our own nine distinctions as they relate to media evaluation projects and our overarching research program here at the USC Annenberg Norman Lear Center’s Media Impact Project.
Nine Key Differences Between Evaluation vs. Research
1. Value: Evaluation focuses on the effectiveness and/or value of a program, message campaign, or other communications; Research strives to be value-free or at least value-neutral in pursuit of increasing knowledge.
2. Role: Evaluators work with stakeholders to understand a program's objectives and goals, and develop agreements on the relevant (and obtainable) "Key Performance Indicators" which constitute evidence these are being achieved. Researchers develop an initial question and design their study (e.g., intervention, experiment) deciding:
what variables will be tested; on whom; under what conditions; over what period of time.
3. Application of Critical Thinking Skills: Evaluators engage as a "critical friend" to program leaders helping to understand analyses, determine effective changes, and refine data collection as the program evolves (a posteriori/ad hoc). Researchers engage in critical thinking at outset of a study to implement procedures which prevent them from biasing data collection or interpretation of findings (a priori).
4. Use and Timing of Operationalization: Evaluators work with project leaders to operationalize terms and agree upon methods at outset of project; however, these may shift, expand, or evolve along with the project. Thus, the process of evaluation is responsive and incorporative of what is learned along the way. Researchers operationalize variables and set methods and procedures at the outset which are to be followed until project is completed.
5. Role in Avoiding Potential Pitfalls: External evaluators may be better positioned to recognize barriers and threats to a program's success, as well as unintended effects. Thus, evaluators often play a role in mediating collaborations between program leaders and key stakeholders to increase effective participation and encourage development of effective procedures. Researchers strive to be mindful of the possibility confounding variables may affect their results in order to exclude or control for these at the outset, or statistically eliminate their effects in analysis.
6. Review of Academic Literature: In Evaluation, purposes for literature reviews vary depending upon stage of program:
7. Purposes and Procedures: Evaluations are conducted to discover strategies and tactics to improve a program. Thus, Evaluation Reports are provided at regular intervals as part of an ongoing process in order to encourage reflection and stimulate discussion among project leaders and key stakeholders to help discover and implement effective adjustments to materials and procedures (e.g., to enhance innovations) while the program is underway. Research seeks evidence to prove a program had the hypothesized effect and/or support a theory, with data analyzed and findings reported in full at the end of the study. Researchers would require approval from an Institutional Review Board to make substantial procedural changes to a research plan while in progress.
8. Dissemination of Findings: Evaluators report findings to program leaders and stakeholders to: provide a record of how the program developed and evolved over time
document programs' effects; help articulate best practices for institutionalizing a program and/or implementing it more widely. Researchers disseminate new knowledge for peer review as a contribution to an ongoing academic narrative.
9. Tables, Charts, and Graphs: Evaluators often get to use more engaging quantitative and qualitative data visualization techniques in reports to clients than permitted in academic journals. Research is reported in academic journals which often limit the permitted number of tables and figures, and most often display only black-and-white or grayscale images.
Thus, at MIP, our evaluations of entertainment projects, documentaries, films and other programs function as a vital component within our overall research mission to study the influence of news and entertainment on viewers. In my next blog, I will discuss how our evaluations of news and entertainment programs play an important role within our research.
By Laurie Trotta Valenti, PhD