How to Ensure Quality Media Analysis
April 14th, 2009Posted by: Hannah Del Porto Posted in Media Measurement, Media Monitoring, Tips, Twitter
There are lots of differing opinions on who is best suited to analyze media coverage. Some say it must be done in-house as company employees are the experts on their own issues and initiatives. Others say PR firms are best because they understand how to track and analyze media and public outreach efforts. Then you have analysts from your monitoring firm who may have the most objective perspective.
Each of these options comes with a unique set of pros and cons. But I believe that the “who” in media analysis is less important than the “how”. Quality media analysis can be ensured through two steps.
1. Make the rules
We have a written protocol for every single client. These instructions outline:
- Features of the system and how to use them.
- A list of article tags.
- An explanation or definition of each tag.
- Instructions on how to apply to the tag.
- Examples of how (and how not) to apply the tag.
We have protocols that are 2 pages long and some that are more than 100. They are a valuable tool regardless of the project’s scope:
- As a reference for analyzing media coverage that is unusual or complex.
- As training material for new media analysts.
- As a reference for performing quality control.
- As a means of communication between client and analyst – so the client understands how their coverage is being analyzed and can provide feedback.
2. Enforce the rules
Quality control on media analysis is essential. Even the best analyst will occasionally misread a mention or click the wrong tag button. So it’s important to have a system in place to catch these mistakes.
Your quality control procedure will depend on a number of variables but, in general, we follow these guidelines:
- Daily quality control checks ensure corrections are timely.
- Documented corrections allow the analyst to review their mistakes, and allow the supervisor to determine if there is an ongoing issue. If a mistake is made repeatedly, re-training on part of the protocol may be necessary.
- Special coverage needs special handling. We keep track of any publication, tag or other coverage attribute that is consistently challenging (or especially important), search these out and ensure thorough quality control.
- More opinions help reduce subjectivity. For “borderline” tag situations, we consult a group of analysts for opinion. This is especially useful on tags such as sentiment where objectivity is more difficult.
Creating guidelines for analysis and ensuring those guidelines are followed will get you pretty far in developing meaningful media analysis. But let me throw a few more buzzwords at you.
1. Training
Analysts need to be experts on their client’s protocol. Depending on the complexity of the coverage and the frequency of tag changes, refresher training may also be warranted. Thorough training also reduces the time required to complete quality control procedures by reducing mistakes.
2. Consistency
The absolute most important quality in a media analyst is consistency. Obviously, it’s important to analyze each mention accurately, but far more important is to analyze each mention as all comparable mentions have been analyzed.
Consistency ensures that you can compare data points – this quarter to last quarter, this year to last year, this year to 5 years ago – and know that you are drawing accurate conclusions.
3. Transparency
The end client should have access to the protocol, the analysis reports and the underlying data. This transparency ensures that the analysts are accurately analyzing the data and faithfully translating the information into reports.
Follow Hannah on Twitter for more rules and/or regulations.
Photo courtesy of: Banalities
Tags: consistency, media analysis, media analyst, protocol, quality control, training, transparency
