How to be a Superhero

 

How to Make Your Research Method the Best it Can be

PILOT STUDIES

Whatever research method you have chosen to employ, you will find that your exact procedure and resources will develop as you take your research forward. No researcher can decide on the perfect procedure on the first draft of their research proposal. Instead it should evolve as more information becomes available. However, once you start collecting data, in most methods it is imperative that the procedure remains constant. To ensure that the optimum procedure has been selected it is essential that a pilot study is performed. Sommer and Sommer (2002, p9) describe this effectively.

“No matter how carefully you reviewed the background literature and designed the study, there is no substitute for a pilot study before the actual data are collected. A pilot study is a preliminary use of a procedure designed to identify problems and omissions before the actual study is conducted. Remember Murphy’s Law – if something can go wrong, it will. Even the best designed experiments will have flaws, and the most carefully developed interview sheet will have ambiguous questions or words that the respondent will misinterpret. There is no way to predict all potential problems in advance. Not even the most experienced survey researcher can compose a perfect questionnaire on the first try. Survey research firms test out their questions in a pilot study before beginning formal data collection. You should plan to do the same in your own research. Set aside a designated time for this after the procedures have been developed. Don’t skip the pilot study in a rush to collect data. A little effort put into it will pay big dividends in improving the precision, reliability, and validity of the data collected in the actual study.”

The pilot study doesn’t have to be a major task; just make sure you get the most from it.  So, for example, if you want to pilot the questionnaire you have designed, get a small sample of people to use it and question them afterwards on their understanding of the questions and answer options.  Alternatively you can ask them to say aloud what they are thinking as they complete the form (they may feel a little self-conscious doing this).  Using their comments you can refine or reword questions, leave out some or put new ones in.

Finally, make sure that you report on this in the Method section of your report.  It doesn’t have to be too detailed.  Something along the lines of:

A short pilot study with 5 participants resulted in the amendment of question 5, the removal of questions 3 and 10, and the inclusion of an additional question (see appendix A for details).

 

Sommer, R. & Sommer, B. (2002) A Practical Guide to Behavioural Research. Tools and Techniques. 5th Ed. Oxford: Oxford University Press

Image: digitalart / FreeDigitalPhotos.net

7 Ways to get people to take part in your research: Part 6

6. Go snowballing

• Expect a low response rate.  A 10% response rate for surveys is normal; 30% is good.  Based on that, how many requests do you need to send out to reach your target sample?

• Don’t rely on a single source for your recruitment.  The more avenues you explore, the more likely you are to reach your sample targets

• If you’ve been putting off asking people to take part in your research or you’ve been waiting a long time for ethical or supervisor approval to start, it can feel like you’ve ticked off another task when you finally do put up those posters.  However, time is often short and you don’t want to waste any more time before discovering that you’ve had a poor response rate.  Pre-empt this by using as many recruitment methods as possible from the start.

• What methods can you realistically use?  This may depend on your target population.  Is this restrictive or as wide as the general population?  Will people be excluded based on age, gender, occupation, language etc?  Make sure your recruitment method is suitable for your target population.

• Recruitment methods include:
  • Posters – where can you (legally) put them to be best noticed by your target population?
  • Direct mail/telephone (if you have postal addresses or phone numbers)  – this can be expensive and you will be expected to include the return postage for hardcopy questionnaires distributed this way.
  • Email – you may need to overcome gatekeepers of mailing lists.  Don’t assume that you will be allowed to send a request to ‘all students’; in fact this rarely happens in most UK institutions.
  • Online social media – blogs, forums, Twitter, Facebook, LinkedIn etc.  This has the advantage of allowing you to include a direct link to your research if appropriate.
  • In person – directly approaching people and asking them to take part can be effective as it makes it more difficult to say no.  This can be time-consuming though, and never put yourself into risky situations/places.

• Finally, don’t forget to go snowballing.  This really is a recruitment method (think of the way a snowball gathers more snow as you roll it along the ground to make a snowman).  It just means you ask your participants to distribute your request to their contacts.  This can work particularly well if you are conducting online research and your participants can send the link to their friends or colleagues.  It also works well if you are using a more restricted population as participants may well have more contacts within this specialist group.

Make it clear in your research report which recruitment method(s) you used and when.

 

 

 

Choosing a Research Method

Once you have decided on your research question and started to read the existing literature on the topic, you must decide what approach you are going to employ to further your research. There are a variety of tools available to you to gather the data you need to answer your research question such as interviews, surveys, case studies, design and test, experiments etc. You will need to decide which tool is appropriate for the research questions and whether you have the skills to use it effectively.

No technique is perfect. For example experimentation suffers from artificiality, and observation may suffer from differences in opinion between observers. However, each tool will have some advantages too such as the economies of scale seen with online surveys. In some instances the most appropriate procedure cannot be used. In this situation a combination of other tools may be the best compromise. Ultimately though, the choice of tool should be decided by the question to be answered and the time, resources and skills available.

Table 1 below is adapted from Sommer and Sommer (2002, p6) and may help you choose the tool appropriate for your research question. For some questions more than one tool may be needed.

Table 1: choosing among research tools

PROBLEM	APPROACH	RESEARCH TOOL
To obtain reliable information under controlled conditions	Test people, equipment or procedures in a laboratory	Laboratory experimentSimulation
To find out how people behave in public	Watch them	Systematic observation
To find out how people behave in private	Ask them to keep diaries	Personal documents
To learn what people think	Ask them	Focus groupInterviewQuestionnaire/survey Attitude scale
To find patterns in written or visual material	Systematic tabulation (categorisation)	Content analysis (qualitative techniques)
To understand an unusual event or specific instantiation	Detailed and lengthy investigation	Case study
To find out what people have done in the past	Check public records	Archival research
To discover whether a function is achievable	Build it in the lab and test against set criteria	Design and test
To generate new guidelines or theories from existing knowledge	Read existing literature and synthesise conclusions or new meaning which may be applied to new situations	Discursive

Sommer, R. & Sommer, B. (2002) A Practical Guide to Behavioural Research. Tools and Techniques. 5th Ed. Oxford: Oxford University Press

Image: Stuart Miles / FreeDigitalPhotos.net

Research Methods Resources

I was recently asked for recommended literature on research methods.  There are too many to put into a single tweet so my current favourites appear here organised according to topic.  Hopefully they will be available in your university library or book shop.  Publishers – if my copies are out of date, or you have a book that you believe will trump mine, I’m happy to receive inspection copies to review!  A couple of online resources worth a mention are: Methodspace and Sage Research Methods.

General Research Methods

Oates, B.J. (2006)  Researching Information Systems and Computing.  Sage: London

Lazar, J., Feng, J.H., & Hochheiser, H. (2010)  Research Methods in Human-Computer Interaction.  Wiley: Chichester, West Sussex

Breakwell, G.M, Hammond, S. & Fife-Schaw, C. (1995)  Research Methods in Psychology.  Sage: London

Fisher, C. (2007)  Researching and Writing a Dissertation.  A Guidebook for Business Students.  Pearson: Harlow, Essex

 

Quantitative Methods

Clark-Carter, D. (1997)  Doing Quantitative Psychological Research. From Design to Report.  Psychology Press: Hove, East Sussex

Kranzler, J.H. (2007)  Statistics for the Terrified.  Pearson: New Jersey, USA

Field, A. (2009)  Discovering Statistics Using SPSS.  Sage: London

 

Qualitative Methods

Silverman, D. (2010)  Doing Qualitative Research.  Sage: London

Richards, L. (2009)  Handling Qualitative Data.  A Practical Guide.  Sage: London

Silverman, D. (2007)  A Very Short, Fairly Interesting and Reasonably Cheap Book About Qualitative Research.  Sage: London

Banister, P., Burman, E., Parker, I., Taylor, M. & Tindall, C. (1994)  Qualitative Methods in Psychology.  A Research Guide.  Open University Press: Buckingham

 

Online Research

Gaiser, T.J. & Schreiner, A.E. (2009)  A Guide to Conducting Online Research.  Sage: London

James, N. & Busher, H. (2009)  Online Interviewing.  Sage: London

 

Writing Up

McMillan, K. & Weyters, J. (2007)  How to Write Dissertations and Project Reports.  Pearson: Harlow, Essex

Clark, S. & Pointon, G. (2009)  Words: A User’s Guide.  Pearson: Harlow, Essex

Marriott, S. & Farrell, B. (1992)  Chambers Common Errors in Written English.  Chambers: Edinburgh

Chambers (1991)  Chambers Effective English.  A Guide to Good Writing.  Chambers: Edinburgh

 

Image: David Castillo Dominici / FreeDigitalPhotos.net

 

7 Ways to get people to take part in your research: Part 5

5. Don’t overdo the images

• Images can help to capture attention and aid comprehension and memory (Levie & Lentz, 1982; Glenberg & Lenton, 1992).

• But images only help if they make sense and aren’t too cheesy.  As nice as that cake is above, what does it add to your understanding of this post?

• You must make sure that you own the copyright or have permission to use the image.

• Don’t give false impressions with your images either.  It can seem like a shortcut to quickly search for the first copyright free photo available, but does the picture really represent the conditions, biscuits, equipment, etc, that you will be using?  A recent poster for a social event in a computer science department depicted a young blonde woman enjoying a cup of coffee – it was definitely not representative of the attendees!

Levie, W.H. & Lentz, R. (1982).  Effects of text illustrations: a review of research, Education & Communication Technology Journal, 30: 195-232

Glenberg, A.M. & Langton, W.E. (1992).  Comprehension of illustrated text: pictures help to build mental models, Journal fo Memory & Language, 31(2): 129-151

7 Ways to get people to take part in your research: Part 4

4. Get out the highlighter

• Highlight key messages in all media: posters, forms, survey sheets, consent forms, information sheets etc.  Unfortunately our desire to include all information ‘just-in-case’ or to cover legal aspects can result in information overload and the most important messages being lost.

• Use headings, boxes and images to lure the eye but don’t overdo it or it will lose impact.

• The front page is more likely to be looked at than the last so make sure the most important information goes here (it can always be repeated on subsequent or the final page).

• If you want someone to take action, put it on the right, separate it out (think Gestalt principles), and put the most important action near the headline or in a different colour.

• Colour can grab the reader’s attention (and animations or sounds in non-print media).  Pay attention to the colour you use; remember to consider issues such as colour-blindness, dyslexia, and cultural meaning.  Kahneman (2011) suggests that bright red or blue is more likely to be believed than duller tones of green, yellow or pale blue.  But remember that too much variation can confuse – your aim is to reduce cognitive load.

• Take care not to highlight the wrong bits.  I once saw a health recruitment poster encouraging people to go for chlamydia testing.  It claimed there would be no prodding, poking, scratching or needles.  Guess which words were ten times larger and bolder than ‘no’!

Kahneman, D. (2011), Thinking Fast and Slow, London: Penguin

7 Ways to get people to take part in your research: Part 3

3. Get Personal

• Use personal language in your recruitment media.  It may not be possible to put an individual’s name on a poster but in subsequent communications it’s a must, particularly if your research requires repeated interactions with the participants.  Attrition (losing participants after the initial sign-up) is more likely and may be more detrimental the more data gathering points you need for each participant.

• Use personal pronouns where possible; you, I, me.

• Have a named individual as a contact point or lead for the research in posters and correspondence rather than just a department or organisation.

• Adding a hand written compliment slip or post it note to printed communications distributed to participants can help (see Garner, 2005).

• If using demographic information to pre-screen people, present the result in a positive way, rather than a non-negative way.  OK, that sounds weird but which of these three phrases would encourage you to continue with the research?  a. “No reason has been found to exclude you from the research”, b. “You may proceed”, c. “Fantastic, you’re just the sort of person we’re looking for” (see Burger & Caldewell, 2011).

Burger, J.M. & Caldwell, D.F. (2011), When opportunity knocks: the effect of a perceived unique opportunity on compliance, Group Processes & Intergroup Relations, 14(5): 671-676

Garner, R. (2005).  Post-It note persuasion: a sticky influence, Journal of Consumer Psychology, 15(3): 230-237

7 Ways to get people to take part in your research: Part 2

2. Reward participation

• Incentives can be used to encourage people to take part but research ethics demand that it shouldn’t be so great that people would take a risk that they wouldn’t normally take.  At what point the incentive becomes ‘so great’ will depend on your population as has been seen historically with the prison population (see Hornblum, 1997 for some examples).

• Rewards can be low cost and for everyone, e.g., refreshments, a chocolate bar, high street shop vouchers (if you have funding!).

• Rewards can be higher value and for the winner of a prize draw, e.g., higher value vouchers (you may even get a prize donated from a local store wanting promotion).  This may require you to gather contact details so consider how to keep this personal data secure.

• Even if you have no reward scheme, your participants should not be out of pocket by taking part in your research.  You may need to supply stamped envelopes if postal return of surveys is required, or provide travel expenses.  Factor in the cost of these during the research proposal and planning stages.

• Make sure the reward follows all ethical, cultural and moral codes and legal requirements.  Remember your participant population: would a reward of a bottle of wine or bar voucher really be appropriate?

• Even if you have no obvious reward, don’t forget to say thank you to your participants for their interest and time.  This could be in person, printed at the end of a form, in a personalised card, or an email.  Remember, they may spread the word about your research and encourage their friends to take part (or the complete opposite).

Hornblum, A.M., (1997).  They were cheap and available: prisoners as research subjects in twentieth century America. British Medical Journal, 315: 1437-1441

7 Ways to get people to take part in your research: Part 1

1. Make it easy to take part

 

• So you like communicating via text?  That doesn’t mean that your participants will.  Have more than one way of letting people contact you, e.g., text, phone, email, Facebook, Twitter.  This can give you more than one avenue for advertising your research too.  Remember that you’ll need to check and synchronise all these contact points though.

 

• If your research needs people to be present in person e.g. take part in a focus group or experiment, have these take place in locations near to the participants.  No-one really wants to travel miles to take part in someone else’s research; why would they?  You might need to have more than one place for the research to take place.  If the research can only take place in one location (e.g., if you need specialist equipment) then make it as easy as possible for the participants to get there, and explain this clearly.  You might need to lay on transport, arrange for people to go together, or pay for travel expenses.

 

• Provide enough information and make it as clear as possible what people need to do to sign up.  Levanthal et al (1965) found that something as simple as providing a map to the room can increase uptake.

 

• Let people sign up straight away. Can they register by signing your poster, clicking on a link in an email, scanning a barcode on their phone to take them to the right URL?

 

Leventhal, H., Singer, R., and Jones, S. (1965) Effectsof fear and specificity of recommendation upon attitudes and behavior.  Journal of Personality and Social Psychology 2(1): 20-29