Abstract: English majors generally are adept at literary criticism but tend to have less experience in conducting empirical research that draws on both qualitative and quantitative methods and engages human participants. To introduce those methods to students and to satisfy a university requirement for quantitative instruction applied to the discipline, I developed a course called Approaches to Research in English Studies. The students complete individual IRB-approved projects that result in a research report, a poster, and a lightning talk. Before undertaking the individual projects, the students engage in a whole-class research project that models the process, and this latter assignment is described here in this essay.
Undergraduate research has been termed the “pedagogy of the 21st century” (Council on Undergraduate Research and National Conferences on Undergraduate Research, 2005, p. 1) and is listed as a “high-impact practice” in undergraduate education (Kuh, 2008). The Association of American College and Universities (2015) has called for students to undertake “signature work” addressing a problem that is important to the student and usually to society; involves substantial writing, research, and reflection; and is likely to be career related. Within the subfields of English Studies—folklore1, English Education2, linguistics, creative writing, technical communication—students can undertake this kind of signature work, conduct empirical research, and draw on qualitative and quantitative methods that cut across all these fields, particularly when text is used as a common thread.
I designed an upper division course, Approaches to Research in English Studies, that enrolls students majoring in English. They complete two research projects: one as a whole group and one as an individual. This essay describes the group assignment that introduces them to empirical research methods and quantitative tools, preparing them to undertake individual research projects. In the group project, they pursue a meaningful research question—developed by the instructor—that gives them practice in applying for IRB approval, designing and administering an online survey, developing and asking interview questions, conducting the study, and publicly sharing their results. Most students enter the course uneasy about empirical research and quantitative data, but through this scaffolded experience involving the entire group, they grow more comfortable. They are then able to move to individual projects that replicate the steps of the larger research project. By the conclusion of the course, according to evaluation results, students feel that they have acquired pragmatic skills and have grown as researchers.
The overarching goal of English 3470, Approaches to Research in English Studies, is for students to learn how to conduct empirical research and complete authentic research projects. By authentic, I mean working on a question of personal interest that has not previously been studied, gathering and analyzing data, and reporting results to a real-world audience. Federal regulations define research as a “systematic investigation, including research development, testing, and evaluation designed to develop or contribute to generalizable knowledge” (U.S. Department of Health & Human Services Office for Human Research Protections, 2018). Key concepts from this federal definition are systematic investigation and contribution to generalizable knowledge. In short, the objective is to add to our knowledge base. A second goal of the course is for students in English to satisfy a university graduation requirement in quantitative understanding and application. Quantitative Intensive (QI) classes are designed so that students learn how numbers apply to their discipline.
The primary assignment of this research methods class is to undertake individual projects that result in a report, a poster, and an oral presentation, called a lightning talk. This is a tall order for a 15-week semester. By undertaking a research project as a group at the beginning of the term, students engage in the research process, which they can then apply to individually selected questions. Both the group and individual projects employ qualitative and quantitative data to answer research questions. The assignment is organized in steps that demystify the process and make research seem less overwhelming:
Find a research question.
Design the study, selecting a method and process.
Review the literature.
Collect data.
Analyze data.
Develop visualizations for these data.
Write up the results in a research report.
For the group project, the instructor defines the research question, and the students generally take a section of the project for literature review and data collection and analysis. Those sections are then compiled in a report, and significant findings of the report are included in a poster, which is presented at a campus undergraduate research symposium. The poster has lead authors, but everyone is listed as investigators. For the individual projects, students do all the steps themselves with instructor advice and support.
Increasingly, students in English, particularly Writing Studies, are engaging in authentic research, responding to calls from scholars in the field who advocate for undergraduate research (Behling, 2010; Dean & Kaiser, 2010; L. Fitzgerald, 2014b, 2014a; Grobman & Kinkead, 2010; Kinkead, 2011, 2016, 2018; J. Kinkead & Grobman, 2011; Kinkead & Moore, 2017; Klos, Shanahan, & Young, 2011; Scott & Meloncon, 2017). The CCCC Statement on Undergraduate Research and its bibliography (2017) demonstrate the significance of student inquiry to the field. The increasing interest in digital humanities (e.g, Cohen, 2010) also provides exciting ways for seeing the field empirically (see Dexter et al., 2017). English is not a one-size-fits-all field. Students who major in English may specialize in a variety of subfields such as English Education, Technical Communication, Creative Writing, Linguistics, Literature, and Folklore. These are the students who enroll in my class.
Beginning with the etymology of the term empirical is useful, as English majors often relish word origins. We discuss Greek physicians who chose to base their practices on systematic observation, accumulating knowledge from practical experience. In short, empirical research means observing and describing what is observed. Students begin to see that as doable. They tend to be unaware that empirically based research can be conducted on a variety of subjects relevant to English Studies. Hudson (2013) argues that words and numbers can be complementary and offers several examples on how quantitative analysis can serve as an important tool in textual analysis. This article can serve as a good source for classroom discussion, but students also need hands on instruction and practice in generating and analyzing quantitative evidence.
Thus, we begin with a relatively easy task, tracking instances of writing for a week: social media, note taking in class, lists, Internet searches, and so forth. During class discussion, we address the hidden complexities, such as what will be measured and how will it be measured: number of instances, amount of time, or volume of text. This assignment also introduces the concept of coding.3 What terms will be used, and are they recognizable and transportable? At the end of the week, results are compiled into a single Excel spreadsheet for an overall snapshot of the class. Then we demonstrate in class how the spreadsheet can be turned into a chart or graph. Some of them literally gasp to see the pie chart or bar graph that appears—almost like magic. Students complete this assignment with increased confidence about their ability to undertake empirical methods and with less resistance to quantitative measures. They say, “That wasn’t so hard,” and “I can do that.”4
The whole-class project is designed to introduce several tools: review of literature, surveys, interviews, archival sources, analysis of data, and graphic depiction of data. Because information will be gathered from human subjects, they become certified through the online CITI modules5. The modules require about two hours of reading and taking quizzes; the certification itself is valid for three years.6 In determining a whole-class project, I choose a topic that will engage as many research tools as possible. Then I begin with my own curiosity. As Zora Neale Hurston (1942) said, “Research is formalized curiosity” (p. 143).
What has been the history of writing instruction on our campus?
How is our Communication Intensive (CI) requirement being enacted across campus?
Are English majors nervous about undertaking empirical research and using quantitative tools?
I will focus here on one group project to illustrate the assignment. When I saw a colleague from the Department of History with a stack of hundreds of blue books to read, it seemed so archaic. As a teacher of writing, I avoid reading handwritten essays. Why would a faculty member undertake such a task? Where did blue books originate in the first place? What do students think about them? A quick Internet search revealed that little, if any, scholarly work exists on the blue book. A gap in the scholarly literature is one of the requirements for pursuing inquiry. We had a project: a comprehensive history and contemporary analysis of the blue book, which engages mixed methods and human subjects7. Our purpose was to shine a light on a ubiquitous but understudied artifact of campus life. My pedagogical purpose, of course, was to introduce multiple methods of inquiry, along with common research tools.
They began with their own experiences through a data dump in free writing. What did they know about blue books? How often did they use them? What were their attitudes toward these essay examination books? They began to see the familiar as novel—an important viewpoint to take as a researcher. In classroom discussion, they asked questions: why are blue books blue, who pays for them, when and where did they begin, is it possible to cheat, why do professors use blue books, what are alternatives, why are some blue books green, who manufactures them, what is the lore about blue books? These questions generated by their own free writing and class discussion set the groundwork for a review of literature. Each student adopted a particular question and began a search for relevant materials.
Through basic Internet research, they discovered that blue book refers not only to examination books but also to other formats, including accepted prices for cars and a list of nobility. But, they also found juicy tidbits about blue book history and contradictory origin stories. Are blue books blue because the rags used to produce the paper came from old Civil War Union soldiers’ uniforms? Or, are they blue because that was the school color for Butler University, purported to be the first higher education institution to use the tool? Another student found a 19th century Harvard professor who railed against written examinations in such blue books, preferring oral tests. What is the future of the blue book? They uncovered fascinating information about student-developed computer applications to replace the examination booklets, concern over who pays for the blue books, the rise of the environmentally sensitive green book, and even fund-raising initiatives that draw on blue book nostalgia. Using Google Scholar, they found that the only academic research on the blue book has been by folklorists (Brunvand, 1986), who reported on the ways students try to scam the system. We looked at databases such as CompPile and also had a class visit to our university archives.
Creating a comprehensive picture of the blue book and practicing as many research tools as possible meant not only conducting a review of literature but also investigating current uses. To do that, we interviewed faculty who use (or do not use) blue books and also surveyed students in classes that take blue book exams. Triangulation is the process of researchers using multiple sources and methods for data collection and analysis (Creswell, 2013). Before we could query human subjects, we needed the approval of the Institutional Review Board (IRB). We wrote the research proposal for IRB8 together, drawing on our review of literature. The proposal is written in an essay format and uploaded to the IRB protocol “app.” The “boxes” of the IRB application software that produces the official protocol uses text copied from this proposal. The application must identify the methods of the study (including survey or interview questions), the process undertaken, recruitment materials, any incentives, and the plan for use of results. All parts of the research process must be defined: how many people will be surveyed or interviewed? What is the breakdown by gender? What return rate for surveys is considered valid in the field? Additionally, letters of information or consent for participation must be part of the application. My local IRB provides templates for these letters. The proposal was approved as exempt “research involving educational tests, surveys interviews, or observations.”
The students individually developed questions to ask informants about blue books. In class, we collated and refined them. (The two sets of questions are included in the supplementary materials.) It is important to learn how to phrase questions9. For face-to-face interviews, as we conducted with faculty, participants should feel comfortable from the start; as a result, initial questions simply gather information about how long they have taught on campus, and the like. Then they are asked about their practices, experiences with type and size of class, cheating, and grading. We did a dry run with a faculty volunteer in class and at the conclusion asked for feedback on the questions themselves. The transcript of that interview offered the opportunity to begin to look for patterns and themes. Interviews require informed consent, a signed document in which the research project is explained.
To gather student-user input, we developed a survey to administer to classes. Although various survey software programs exist, I tend to favor Qualtrics10. Again, students drafted questions, and then we designed the survey during class to demonstrate and explain survey construction. Does the question ask for a yes/no answer? Or is it more appropriate to use a Likert-type scale? What about short answers? This exercise helps students understand the ways questions can be framed and what the implications are for analysis. Short answer responses will result in thicker, more qualitative answers but require more time for analysis. Qualtrics also provides visual representations of responses for easy-to-understand results. We also discussed the response rate and what is credible for an online survey that is distributed by e-mail. Our survey dry run was by class members themselves. Questions and processes were modified as a result. I cannot emphasize enough how these dry runs build confidence among the researchers. The informed consent document was embedded at the beginning of the survey, offering the opportunity to continue or opt out.
Additionally, we wrote the recruitment scripts—yet another new genre for them—to invite participants to the study11. (See supplementary materials for sample recruitment scripts.) Incentives to participate must also be included in the IRB application. For the survey of students, we offered ice cream12 to the class with the largest percentage of participation.
Participants in any study must consent to be involved. Our institution uses two different forms for informed consent. In one, participants remain anonymous, and in the second, it is signed. The decision on which form to use creates a lively classroom discussion as acquiring any identifiable information from participants increases the difficulty of maintaining confidentiality and protecting privacy. They also learn from the IRB about how information gathered from informants is to be kept protected in locked files in locked offices or on secure online environments. In this particular case, interviews required signed consent, and surveys were anonymous. In sum, the IRB proposal requires that the project be thoroughly explained and contain all of the documents and procedures that will be used in the study. This is a formidable, but doable, amount of detail to be created.
Once IRB approval was received, the students worked in pairs to interview faculty members about their use (or non-use) of blue books and then invited their classes to participate in the survey. The results of the interviews were anonymized and shared among the class so that we could look for themes and patterns. Small groups looked at all materials and then reported on their findings so that we could come to whole-class consensus. This information could have been shared over our course management system, the secure Box system, or Google docs. I chose to distribute copies in class and then collect them at the end to ensure privacy of our informants. We also reinforced what we came to call the “What happens in Las Vegas, stays in Las Vegas” principle that we did not discuss our research project results beyond class investigators. The survey responses were anonymous by design. We sent a second email invitation for participation to boost the response rate. And, one class celebrated with ice cream for winning the competition. For all classes and faculty participation, notes of gratitude signed by all researchers were distributed.
With evidence in hand from the review of literature, interviews, and surveys, the team members adopted various sections of the final report to write: the research question, review of literature, methods, findings, limitations of the study, and suggestions for further research. A standard structure for research reports is the IMRAD (or IMRaD) format, an acronym for Introduction, Methods, Results and Discussion. The report describes what was done and what it means—why it is significant—typically ending with a note about what future research might be undertaken. The researcher is also frank about any limitations of the study. A novel concept for English majors, limitations appear at first glance to show weakness. This genre requires more of a “just the facts” tone but also draws on students’ narrative abilities to tell a good story.
An important section of this assignment is sharing research. Drawing on the research report, we drafted the research poster as a class. A group of two or three students volunteered to develop the poster, which the class then critiqued for revision. The final product was delivered at a campus undergraduate research day with all students listed as authors and the volunteers taking the role as first authors. The lightning talk developed in a similar way with students designing a storyboard of the talk in class and then a few developing the slides, drawing, for instance, on the visualization of graphics developed from the Qualtrics survey. It is the rare student who does not contribute in some important way to this collaborative effort. These various genres introduce rhetorical concepts of purpose, audience, style, and organization, and, it is helpful to talk about each as a genre with specific conventions. The blue book report proved to be sufficiently worthy and interesting to submit to a peer-reviewed journal. It was condensed to 17 pages (including survey instruments) and eventually accepted for publication (Duersch et al., 2018). Two students took the lead on that project with one of them finalizing the published version, but all received credit as authors. In sum, they experienced processes that a researcher enacts from concept to dissemination. That is a hallmark of the authentic research project.
The whole-class project provides the framework for the independent projects. The work involving the entire class begins very quickly at the start of the term; the independent projects take off shortly thereafter, tracing the steps of the larger endeavor. Imagine two trains going down parallel tracks. The whole-class project train is some lengths ahead with the individual project train running slightly behind. The whole-class project pulls into the station about two-thirds of the way through the semester.
In terms of evaluation, the independent project receives the bulk of the points for coursework, 250 points divided among each task. An abstract might be 5 points while the comprehensive report is 100. For the whole-group project, I tend to offer more in the way of participation points, say 5-10 points for students submitting interview questions or writing a section of the comprehensive report. Peer input and the desire to learn the research process for their own independent projects keep students on task and engaged.
Shortly after the group project launches, students choose three questions about which they are curious. Their diverse interests never fail to astound:
A technical writer wondered why there seems to be a rise in expensive calligraphy on wedding invitations when digital production is less costly;
A creative writer wanted to know what parents look for in faith-based children’s literature;
A gamer wondered about players’ awareness of grammar, usage, and mechanics in Massive Multiplayer Online Role Playing Games (MMORPGs);
A future teacher asked if ethnicity and gender and their representation in school district-approved reading lists matched the demographics of the student population;
Another future teacher had heard negative feedback about the introduction of a 1:1 laptop program in a secondary school.
These points of entry determined the research approaches. The whole-class project used mixed methods, but they learned about alternative approaches from reading in the professional literature.
Working with undergraduate researchers in English led to my book, Researching Writing: An Introduction to Research Methods (Kinkead, 2016). It is the first single-authored textbook on conducting research in writing studies directed at undergraduates. As an advocate of undergraduate research, I had been frustrated with the lack of such a text that provided foundational information and practice of research. Researching Writing covers a range of approaches to research: ethnography, case study, textual analysis. Each approach is exemplified by published examples of undergraduate research, primarily derived from the journal Young Scholars in Writing (YSW)13 but also from other appropriate sources such as The Writing Center Journal (Brown, Fallon, Lott, Matthews, & Mintie, 2007). Using published student work sends a message that it is possible for students to join the scholarly conversation. Moore (2017) says that Researching Writing “presents complex ideas about research in digestible chunks and intersperses them with engaging activities to facilitate both individual reflection and whole-class or research-team discussion.” The two major assignments in the course require students to undertake research that is replicable, aggregable, and data supported (Haswell, 2005) and has the potential to contribute to the field.
Having taught this assignment for several semesters, I am learning that the whole-class project is essential but needs to be sufficiently compact so that we can complete it as expeditiously as possible. We have analyzed Communication-Intensive classes and reported to the University General Education Committee (Kinkead, 2018); we have studied English majors’ perceptions of empirical research and quantitative data; we have compiled a history of writing at our institution from its beginnings in 1890 to be placed in the University Archives. These are ambitious projects, and I feel the need to scale back somewhat to make the assignment more manageable. Each semester is a challenge, as it requires a new area of inquiry in order to meet the criterion of authentic research that adds to our knowledge base.
One of the outcomes of the course assignment is the concrete skill set that students acquire. They are certified in human subjects research and know how to construct an online survey, interview informants, and depict information graphically14. They have developed new genre expertise by communicating research in three separate formats. Consistently, after this comprehensive assignment, students note their increased comfort level with empirical research, quantitative information, research ethics, and public speaking. They are not put off by quantitative information, and they think critically about statistical reports and how graphic information is depicted. They report this growth in qualitative comments not only on end-of-term evaluations but also through pre-and post-surveys that measure their comfort level with empirical research and quantitative data tools. At the beginning of the term, a survey revealed that 90 percent did not feel confident about their understanding of empirical research. By the course’s end, they were more assured about their role as “systematic observers.” As one student, who analyzed literary memes on social media, wrote on an end-of-term evaluation, “I see research questions everywhere now.”
At the beginning of the course, 60 percent did not feel comfortable using quantitative tools. By term’s end, a student noted how figuring mean, median, mode, and range of results gave her multiple ways to analyze information and to be a “statistics skeptic.” Students used terms such as “daunting” to describe their initial reaction to conducting empirical research but ended with “making conclusions about data and then finding ways to graph them was one of the most gratifying products of this process.” Yet another student who was curious about gendered pronouns in Dungeons & Dragons handbooks over several decades, noted, “I have made huge leaps in my proficiency, understanding, and comfort when it comes to quantitative data projects. . . . I had zero experience with quantitative studies. Now I want to undertake a larger study on my own free time.”
These students are not cowed by data but able to make data-informed decisions. They see themselves as researchers. They have begun to advance on the scholarship continuum that Grobman (2007, 2009) and others (Perdue, Driscoll, Matthews, Paz, & Tess, 2014) noted, and they are more prepared for graduate studies or a career, developing projects that have the capacity to influence and inform the profession. As an early-career secondary school English teacher, who has started a master’s degree program, wrote: “Yesterday I had my first graduate education research course. I felt as if I had been transported back to your researching writing course! I felt so pleased to know that I was familiar with almost all of the terms we covered on the first day.”
For examples of student work (posted with permission and/or IRB approval), see researchingwriting.weebly.com.
(To view a PDF facsimile of the original formatting of this assignment, scroll to the top, look to the right-hand Article Tools, and click the Supplementary Files link.)
During the course of the semester, you will complete an empirical research project on a topic in English/Writing Studies. This should be a research question that can be answered in the confines of the semester. It will require a review of literature to ensure that it is a topic that will fill a gap in the scholarly knowledge base. You will choose a method appropriate for that question, then design and conduct the study, drawing on both qualitative and quantitative evidence. The research question is closely related to how the question is addressed, and thus the research method selected is an important decision. We will begin with a whole-class research project that anticipates the steps that you will undertake for your individual projects and gives you practice.
In summary, you will do the following:
Find a research question
Design the study
Review the literature
Collect data
Analyze data
Develop visualizations for the data
Write up the results in a research report
Present the findings in two additional formats: a lightning talk and a research poster.
The parts of the individual research project include the following:
Research Question
Annotated Bibliography
Review of Literature
Research Proposal
Certification in research integrity through CITI Program
IRB Application (including recruitment script and materials, Informed Consent forms)
Abstract
Research Study Report
Research Poster
Presentation of research poster
Oral Presentation
Delivery of Oral Presentation
You will begin by identifying three areas or questions of interest to you in English/Writing Studies. From your Writing Autoethnography assignment in Week Two, you probably have already discovered areas of interest. That's a good start. Another possibility may focus on a question you've always wondered about in English/Writing Studies. This assignment begins to develop the research question or problem that will be the focus of the project. In a one-on-one conference during Week Two, we will explore these questions. Almost always, we can derive a viable research question from our conversation. Once this question is developed, then the review of literature can begin that then leads to the methodology development and study.
What is it you want to explore or know? What question will help you find the answers? What will you study in order to get answers--people, texts, artifacts, practices? Why is this question important or significant? How will it make a contribution to the field, and, finally, is it doable?
A review of literature is the scholarship that has been done already on the research question, either on the question itself or closely associated. This review helps the researcher identify what has been done, how it was done (methods), and what the results were. It also suggests that there is a gap that your project will fill. Develop an annotated bibliography (Week Three), which will be used to frame the review of literature (Week Four). Note that review of literature is the process as well as one of the products that will result from your work. The annotated bibliography is a listing of sources with an annotation for each; the review of literature takes that information and synthesizes it into more of an essay form.
A review such as this is invaluable in helping the researcher (you) figure out the actual study, defining it more clearly. A good review will affect your thinking about the topic, perhaps tweaking or even abandoning the topic--particularly true in the latter case if the researcher finds the question has already been addressed fully. This review of literature becomes part of the final project. (The same is true of many parts of the process. Each section is compiled into the final report.)
Complete CITI Certification (valid for three years). This is an important part of Responsible Conduct of Research (RCR) that focuses on the history of ethics in research and also guides our studies today.
Develop the Research Proposal (Week Five), which will be embedded in the IRB application (Week Six). This text will be direct and to the point. Begin with the Project Title. Then state the research question. Provide 2-3 sentences that clearly state the purpose and significance. Then, the review of literature cites the scholarly/research literature that has already been done. Its purpose is to demonstrate that you know the background, and a gap has been identified—that your research will fill. Provide a step-by-step explanation of the methods you will use. What are your sources for information (e.g., people, texts, artifacts, archives)? What research tools will you use (e.g., survey, interviews)? If people are involved, describe your participants. Who are they? What ages? How will you locate them? How will you recruit them to your study?
Our research projects will use surveys and interviews to gather information. Develop the questions for each. For the surveys, we will use Qualtrics and deliver the survey directly to the identified population for completion. Qualtrics provides reports that help depict the quantitative data in the research report. For interviews, the transcription of the interchanges will be coded for themes and patterns.
Conducting the study will take place over the next several weeks; the start time is dependent upon IRB approval. By Week Eleven, the draft of the research report should be completed.
Below is an outline of how the study and its data will be presented. A common term for the organization of the final report is IMRAD: Introduction, Methods, Results, and Discussion.
Study Name
Abstract
Introduction and problem statement: Give an overview of the context surrounding the research, and a statement of the research question.
Review of the literature: A review of literature of previous studies that have focused on this question is included here, and how this study extends a previous project or fills a gap is defined and explained.
Participants and context of the investigation: Describe the participants in your study in terms of who was selected, how they were selected, demographic information.
Method and Procedure: Describe the method you used for gathering information. This may be one method, or it may be a mixed methods approach.
Analysis and Results: Present the data15 that have been collected; then analyze them according to the method you’ve selected.
Discussion: Synthesize the results in the context that has been defined in the introduction. What do these results mean? Are there implications for practice or policy? What is the significance in terms of your initial research question? What future research does it call for?
Graphic Representation of Information: Include illustrations, charts, and tables that depict information graphically and demonstrate good principles of design.
The research report is one of three genres used to give the results of the study. We will use two additional formats: a research poster and a lightning talk.
Poster Development: Design a poster in PowerPoint that summarizes the research project and illustrates data graphically. We will draw on design advice for effective posters (Week Fourteen).
Lightning Talk: Develop a lightning talk of no more than 3-5 minutes with 15-20 slides maximum (Week Fifteen).
IRB Proposal
Recruitment script by telephone inviting faculty to participate
Interview Questions: Faculty
Recruitment script by email inviting students to take survey.
Survey Questions: Students (with Letter of Information)16
Research Proposal
An Examination of Examination Booklets: The Blue Book
Introduction:
The Blue Book is the standard for essay examinations and has been used extensively since its introduction at Butler University in Indianapolis. According to some sources, it is blue because Butler University’s school colors are blue. Butler University, a liberal arts school, was founded in 1855 by attorney and abolitionist Ovid Butler “on the principle that everyone deserves an excellent education, regardless of race or gender—a bold idea in pre-Civil-War America.” A competing origin myth is that Harvard introduced the blue book in 1857 with Yale following in 1865 (Weiss). This mass-produced paper product assured that students would have an inexpensive resource for taking essay exams and that all students’ work would appear in a format that was consistent and fair.
In some arenas, the Blue Book is loathed. Take this advice from Cal Newport, a professor at Georgetown University, who writes “Study Hacks Blog”: “As we tumble toward final exams, I’d be remiss if I didn’t address one of the most dreaded denizens of the season: the blue book essay exam. Nothing strikes more fear into the heart of a liberal arts student than seeing that big blue book, full of empty, lined pages, just waiting to be filled with paragraphs pregnant with novel insight.” A similar view of blue books appeared in the University of Virginia magazine, “Blue books: The agony and ecstasy of final exams.” But sales figures of Blue Books at UVA are about 17,000 blue books and 10,000 green books annually.
Roaring Spring Paper Products of eastern Pennsylvania has been producing blue books since the 19th century. Pontiac Paper Company from which Yale purchases most of its supply (Marberg) produces 8-9 million blue books per year, and even though it touts itself as the "birthplace of blue books, it did not enter the market until 1976. Testing Forms (http://www.testingforms.com/store/testing-booklets-c-28.html) of Oregon is another supplier.
In recent years, there has been a sense that the Blue Book is diminishing in popularity for several reasons: electronic testing, concern over sustainability issues, and problems with cheating. For instance, students at the University of Pennsylvania developed a blue book substitute app: Emerald Exam, a desktop application that allows students to take written exams on a computer. The “emerald” may refer to the more sustainable profile of an e-version of an exam. The “Green” Book, sold by some paper companies results from recycled paper. Cheating scandals with re-sold or re-used blue books resulted in campuses varying the colors on a semesterly basis (Marberg).
To date, no scholarly study has been undertaken to ascertain if the decline of the Blue Book is true. We envision a final product tentatively titled “The Rise and Fall of the Blue Book,” but need data to test whether that hypothesis is accurate. The proposed title is an allusion to a landmark study in writing studies by Robert Connors, “The Rise and Fall of the Modes of Discourse.”
Methods:
Students in English 3470, Approaches to Research in English Studies (n=18), along with their professor who functions as PI, will investigate several aspects of Blue Books that draw on multiple methods and research tools:
Historical/archival method will be used to investigate the history of the Blue Book, where and how it originated, and why it became the standard instrument for written examination.
Content, textual, and rhetorical analysis will be used to interrogate the Blue Book itself as an artifact. Although the standard examination book is blue, “green books” became more popular with a focus on sustainability. Blue Books vary somewhat in size and appearance.
Case study method will be used to analyze the uses of the Blue Book by a selected few faculty members. The PI will contact faculty in target departments (most likely humanities and social sciences, which have a tradition of using blue Books) and then select a few representative faculty for interviews predicted to last no more than 30 minutes. The interviews pose minimal risk to participants and will focus entirely on the participants’ philosophy about using blue books, their advantages and disadvantages, and their predicted continued use as an instrument for testing.
For student participants, a survey will be administered to one or two classes taught by faculty members who have participated in interviews. The survey will be brief and query students about their experiences with Blue Books.
Statistical information about sales will be drawn from the USU Bookstore and used as evidence to determine if Blue Book usage is declining, remaining the same, or increasing.
It may be necessary to query faculty at other institutions, particularly eastern liberal arts colleges/universities, as it is entirely possible that the blue book is consumed differentially among institutions. If we pursue this avenue, we will contact faculty with whom we are familiar and also use an Informed Consent.
No video, photo images, or audio records will be made.
Target Population:
Faculty who use or have used blue books for essay examinations and are employed by USU. They will be approached by the PI who will recruit by telephone. For faculty, a small convenience sample should elicit the reasons behind their use of blue books. We estimate that 5 male faculty and 5 female faculty will be invited. We anticipate that 3 male and 3 female faculty members will participate.
Students are a second target audience. We estimate about 100 USU students, equally divided between male and female (according to the university’s demographics) will be invited to participate. We expect 40 male and 40 female students to participate. We will exclude anyone under the age of 18.
Incentives:
The incentive to complete the Student Survey (Qualtrics) will be a prize for the group of students (a single class) that has the highest percentage of completion (what percent of class enrollment completed) on the survey within three days' time of receiving the invitation to participate: Aggie Ice Cream for the class members and their teacher. There is no compensation of incentive for faculty participants (although the instructor of the winning class will benefit from ice cream, too).
Anonymity and Confidentiality
Information derived from faculty by interview will be anonymized, and student participants will be anonymous. In the case of the latter, the survey will be administered via Qualtrics. In the case of faculty participants, identifying markers will be removed and replaced with an identification number. All information will be kept secure in a locked office in locked filing cabinet, and any electronic materials will be kept on a password-secure computer using the USU Box system. Participants may withdraw from the study at any time.
Results:
Student investigators will compile their findings into a research report. The overarching purpose of this research proposal is to teach student researchers how to conduct research, especially in English 3470, Approaches to Research in English Studies. This is not just academic research. The intent is for the students to carry the project through to dissemination, and the journal, Young Scholars in Writing: An Undergraduate Journal in Research in Rhetoric and Composition is the target publication.
Conflict of Interest and Responsible Conduct of Research:
Neither the PI nor the student investigators have a conflict of interest. The PI has the ultimate responsibility for the protection of the rights and welfare of the human subjects, the strict adherence to IRB requirements, federal regulations, and state statues. The PI assumes responsibility for ensuring the competence, integrity, and ethical conduct of the student researchers and certifies that all researchers have current IRB/CITI certification. No changes will be made to the study without approval by the IRB. The study will be closed out through the IRB at its conclusion.
Sources:
About Butler University, History and Traditions. https://www.butler.edu/about-butler.
About Roaring Spring Paper Products. http://rspaperproducts.com/about/. Accessed 4 September 2016.
“Blue Books: The Agony and Ecstasy of Final Exams.” University of Virginia Magazine. Spring 2013. http://uvamagazine.org/articles/blue_books.
Bhanpurawala, Juzar. “Cheaters How to Cheat: Techniques Used by Cheaters.” 12 March 2012. https://www.facebook.com/notes/juzar-bhanpurawala/cheaters-how-to-cheat-techniques-used-by-cheaters/339597816087009.
Connors, Robert J. “The Rise and Fall of the Modes of Discourse.” College Composition and Communication 32.44 (Dec., 1981): 444-455. http://www.jstor.org/stable/356607.
Courtney, Jennifer. “How Blue Books Get at the Heart of Assessment.” Classical Conversations, Classical Christian Community. 6 March 2014. https://www.classicalconversations.com/article/how-blue-book-exams-get-heart-assessment.
Galica, Gregory S. The Blue Book: A Student’s Guide to Essay Exams. NY: Harcourt, 1991.
Guess, Andy. “Beyond the Blue Book.” Inside Higher Education, 17 October 2007. https://www.insidehighered.com/news/2007/10/17/testing.
Hallett, V. C. “Blue Book Birthplace.” Harvard Crimson, 8 April 1999. http://www.thecrimson.com/article/1999/4/8/blue-book-birthplace-pgilroy-calif-is/.
Hatch, Robert A. “Like, What’s a Blue Book, Dude?” February 2001. http://users.clas.ufl.edu/ufhatch/pages/02-teachingresources/readingwriting/05bl-bk.htm.
Kessler, Zara. “How to Stop Cheating at Harvard (and Yale).” Bloomberg View, 4 September 2012. https://www.bloomberg.com/view/articles/2012-09-04/cheating-at-harvard-yale-etc-.
Marberg, Sarah. “Why Are Blue Books Blue?” Yale Daily News, 17 November 2004. http://yaledailynews.com/blog/2004/11/17/why-are-blue-books-blue/.
Mikkelson, David. “Book of Daze.” 20 June 2011. http://www.snopes.com/college/exam/lostbook.asp.
Nadig, Carl. “Pass/Fail: Blue Book Exams Save Print Fees.” Northern Star (Northern Illinois University) 1 October 2014. http://northernstar.info/opinion/pass-fail-blue-book-exams-save-print-fees-protest-bombs/article_049cc224-493f-11e4-935e-001a4bcf6878.html.
Newport, Cal. "How to Ace Essay Questions Using the Three-Minute Rule. 8 December 2008. Study Hacks Blog. http://calnewport.com/blog/2008/12/08/how-to-ace-essay-questions-using-the-three-minute-rule/
Powner, Leanne C. “Maximizing Your Grade on Blue Book Exams.” Toolbox. 2007. http://jayandleanne.com/lpowner/tchdocs/bluebooks.pdf.
“Quick Question on the Blue Book.” [Faculty avoiding cheating.] College Confidential. 18 May 2009. http://talk.collegeconfidential.com/college-life/716571-quick-question-on-the-blue-book.html.
Smola, Jennifer. "Could Blue Books Be Coming to an End? Penn Students Develop Paperless Exam Alternative. USA Today, 12 December 2012. http://college.usatoday.com/2013/12/12/could-blue-books-be-coming-to-an-end-penn-students-develop-paperless-exam-alternative/.
Valentine, Paul W. “Honor Among Scholars.” Washington Post, 1 May 1994. https://www.washingtonpost.com/archive/1994/05/01/honor-among-scholars-thrives-out-of-uniform/bea645eb-9a33-4654-be2c-1d70f7b89853/.
Weiss, Kenneth R. “Dreaded Blue Books May Go the Way of Quill Pens.” Los Angeles Times. 12 March 1999. http://articles.latimes.com/print/1999/mar/12/news/mn-16486.
Wolchek, Scott. “Mouthing Off: Blue Book Blues.” The Oakland Post, 2 March 2014. http://oaklandpostonline.com/4023/satire/mouthing-off-blue-book-blues/.
Recruitment Telephone Script for Faculty
[This is a telephone script to recruit faculty to the Blue Book study. Telephone calls will be made to faculty in the humanities and social sciences, who have a tradition of using Blue Books for essay examinations. I will be using a convenience sample, calling colleagues who know me.]
Hello, this is Joyce Kinkead in the Department of English.
My students in a new Quantitative Intensive (QI) class, ENGL 3470, Approaches to Research in English Studies, are undertaking a study of the Blue Book, the standard examination booklet for essay tests. We are extending an invitation to USU faculty in disciplines that have a tradition of using blue books to participate in this study.
Do you use blue books or have you used blue books in the past?
[Decision tree: No results in a message of thanks and sign off.]
[Decision tree: Yes results in a continuation of the script.]
Please know that the focus of the research is on the Blue Book itself as an essay exam instrument; it does not center on individual teachers and is not evaluative in any way. Let me tell you what would be involved so that you can make an informed decision. Two students from the class will visit you at your convenience in your office for an interview to ask questions about your use of blue books. The interview is predicted to take no more than 30 minutes.
Would you be willing to discuss your use of blue books as an examination tool?
[Decision tree: NO results in a message of thanks and closure of the conversation.]
[Decision tree: YES results in continuation of the script.]
That is great. Can you tell me your office hours or convenient times? And your office address?
[Take note of preferred times.]
The students will give you a formal letter of consent to sign when they visit you in your office. Let me take some time here, though, to note how we as researchers will protest your privacy and keep your answers confidential. We will use identifiers to preserve anonymity, replacing all names. Research records will be kept in a secure location and destroyed at a later date. At any point in the study, participants can withdraw without penalty. Let me reiterate that point: you may choose to withdraw at any time.
Do you have any questions about the study?
The students in English 3470 are learning important skills in undertaking research.
Thank you for your contribution.
Good-bye.
[Conclude telephone conversation.]
Interview Questions: Faculty
Blue Book Questions for Faculty – Interview
Review Informed Consent document and get signatures (leave one copy with the faculty member and give one copy to Dr. Kinkead). Student researcher names/contact information the letter prior to arrival at interview.
How long have you been a professor?
Do you use blue books for examinations in your classes? What are their advantages? Their disadvantages?
If you once used blue books for examinations but no longer do, why did you change your approach? What were their advantages? Their disadvantages?
If you have never used blue books for examinations, why did you make that choice? What were their advantages? Their disadvantages that outweighed any advantages.
How often do you use them?
Please describe how you use blue books for examinations.
Do class sizes vary and have an influence on your use of blue books?
Do you use blue books in general education classes? Majors classes? Both?
Do you provide the blue books for the students? Or do they bring their own?
Do you feel that blue books are readily available on campus for students?
What do you do if a student has forgotten to obtain a blue book?
Blue books come in different sizes and formats. Do you specify a particular size?
Do your students tend to use all pages of the blue book in completing their examinations? Is there a range? Or, do you suggest a length for their responses?
Cheating can happen with any assessment. Can you tell us about any instances of cheating with blue books?
Do you take any measures to prevent cheating with blue books?
How do you handle handwriting that is illegible?
Would you favor word-processed essay examinations if they could be administered in a secure environment?
What is the typical amount of time to evaluate blue book examinations and inform students?
What happens to completed and graded blue books? Do you keep them? Return them to students? Do students pick them up?
If you could recommend any changes to the blue book format, what would it be?
Did you use blue books when you were a student? What was your opinion of them then?
Blue books have been a traditional assessment tool since the mid-nineteenth century. What do you see as their future? Do you think they have staying power?
Is there anything about blue books that we have not asked that you’d like to share with us?
Will you be using blue books this semester for an exam that would occur around midterm?
If yes, then issue the following invitation:
Request for Student Participation:
In addition to talking with faculty about their use of Blue Books, we’d like to gather information from students, too. We have a brief Qualtrics survey that should take no more than 10 minutes for students to complete. Ideally, the students would complete it within a few days of using a blue book for an examination. Students would receive an e-mail link to the Qualtrics survey.
We are hoping that you might agree to have your students consider taking this survey. We are inviting a few classes to participate, and there is an incentive prize: the class with the highest percentage of participation will receive free Aggie ice cream.
Would you be willing to have your students participate in this survey? Please know that it is entirely optional, and you may withdraw your students’ participation at any time.
If Yes: Thank you. Do you know the date when your students will be using blue books? We will be back in touch to finalize details.
Thank you!
Script sent to students with survey link
[E-mail that will introduce the Qualtrics Survey on Blue Books]
Hello,
This is Professor Kinkead. I introduced myself during your class. The students in English 3470, Approaches to Research in English Studies, are studying the use of Blue Books for examinations. Because your class is using blue books for essay tests, we are inviting you to complete a survey about their use.
Please know that you are not required to complete the survey and that you may only participate if you are 18 or older. We are inviting a very few classes to complete the survey, and the class with the highest percentage of completion within three days following the email link to the survey will be treated to Aggie Ice Cream.
The survey itself will take no more than 10 minutes of your time. You will be helping the members of English 3470, Approaches to Research in English Studies.
There is minimal risk to you for completing this survey. All results will be kept confidential, and your answers will be anonymous. Results of the survey will be kept in a password-secured environment. You may choose to withdraw from participation at any time without any penalty.
We hope that you will consider accepting this invitation in our investigation of Blue Books. This is the link to the survey.
Thank you
Survey Questions: Students (with Letter of Information fronting the survey)
Blue Book Questions for Students (Qualtrics Survey)
American Council of Colleges and Universities. (2015). Signature work. https://www.aacu.org/sites/default/files/files/LEAP/LEAPChallengeSignatureWork.pdf.
Anderson, P., Charney, D., Cooper, M., Kirklighter, C., Mortensen, P., & Reynolds, M. (2001). Guidelines for the ethical treatment of students and student writing in composition studies. College Composition and Communication, 52(3), 485–90.
Behling, L. (2010). Reading, writing, & research: Undergraduate students as scholars in literary studies. Washington, DC: Council on Undergraduate Research.
Brown, R., Fallon, B., Lott, J., Matthews, E., & Mintie, E. (2007). Taking on Turnitin: Tutors advocating change. The Writing Center Journal, 27(1), 7–28.
Brunvand, J. H. (1986). The Mexican Pet: More "new" urban legends and some old favorites. Norton.
Cohen, P. (2010, December 3). Analyzing literature by words and numbers. New York Times, p. C1.
Conference on College on Composition and Communication. (2017). CCCC position statement on undergraduate research in writing: Principles and best practices. cccc.ncte.org/cccc/resources/positions/undergraduate-research.
Council on Undergraduate Research and National Conferences on Undergraduate Research. (2005). Joint statement of principles in support of undergraduate research, scholarship, and creative activities. http://www.cur.org/about_cur/history/joint_statement_of_cur_and_ncur/.
Creswell, J. W. (2013). Qualitative inquiry and research design: Choosing among five approaches (3rd ed.). Thousand Oaks, CA: SAGE Publications.
Dean, J. M., & Kaiser, M. L. (2010). Faculty-student collaborative research in the humanities. CUR Quarterly, 30(3), 43–47.
Dexter, J. P., Katz, T., Tripuraneni, N., Dasgupta, T., Kannan, A., Brofos, J. A., … Chaudhuri, P. (2017). Quantitative criticism of literary relationships. Proceedings of the National Academy of Sciences, 114(16), E3195–E3204. http://doi.org/10.1073/pnas.1611910114
Duersch, M., Blevins, D., Bennett, M., Bess, G., Bitner, J., Dotson, J., … Watson, N. (2018). The rise and fall of the blue book: An examination of essay exam books. Young Scholars in Writing, 15.
Fitzgerald, L. (2014a). Undergraduate research in writing studies. In I. Crawford, S. Orel, & J. Shanahan (Eds.), How to get started in arts and humanities research with undergraduates (pp. 94–106). Washington, DC: Council on Undergraduate Research.
Fitzgerald, L. (2014b). Undergraduate writing tutors as researchers: Redrawing boundaries. The Writing Center Journal, 33(2), 17–35.
Fitzgerald, W., & Midiri, N. (2013). But is it really research? Undergraduate anxieties in the humanities. PURM: Perspectives on Undergraduate Research Mentoring, 2(2), 1–13.
Gabbert, L. (2010). Exploring local communities: Conducting ethnographic research in folklore studies. CUR Quarterly, 30(4), 37–42.
Grant-Davie, K. (1992). Coding data: Issues of validity, reliability, and interpretation. In G. Kirsh & P. A. Sullivan (Eds.), Methods and methodology in composition research (pp. 270–286). Carbondale, IL: Southern Illinois University Press.
Grobman, L. (2007). Affirming the independent researcher model: Undergraduate research in the humanities. CUR Quarterly, 28(1), 23–28.
Grobman, L. (2009). The Student scholar: (Re)Negotiating authorship and authority. College Composition and Communication, 61(1), 175–196.
Grobman, L., & Kinkead, J. (2010). Undergraduate research in English Studies. Urbana, IL: National Council of Teachers of English.
Haswell, R. H. (2005). NCTE/CCCC’s recent war on scholarship. Written Communication, 22(2), 198–223. http://doi.org/10.1177/0741088305275367
Hudson, P. (2013). Numbers and words: Quantitative methods for scholars of texts. In G. Griffin (Ed.), Research methods for English Studies (2nd ed., pp. 133–159). Edinburgh: Edinburgh University Press.
Hurston, Z. (1942). Dust tracks on a road. J.B. Lippincott.
Janangelo, J. (2010). Anatomy of an article: How an undergraduate researcher turned a passionate project into a published essay. Kairos: A Journal of Rhetoric, Technology, and Pedagogy, 16(1).
Kinkead, J. (2011). Undergraduate researchers as makers of knowledge in composition in the writing studies major. In R. Gebhardt & L. Massey (Eds.), The Changing of Knowledge in Composition (pp. 137–160). Logan, UT: Utah State University Press.
Kinkead, J. (2016). Researching writing: An introduction to research methods. Logan, UT: Utah State University Press.
Kinkead, J. (2018). Engaging undergraduate researchers in the assessment of communication across the curriculum courses. Across the Disciplines, 15(2), 15–30.
Kinkead, J., & Grobman, L. (2011). Expanding opportunities for undergraduate research in english studies. Profession, 218–230. http://doi.org/10.1632/prof.2011.2011.1.218
Kinkead, J., & Moore, J. (2017). The Naylor Workshop in undergraduate research in writing studies. Literacy & NCTE. http://www2.ncte.org/blog/2017/09/naylor-workshop-undergraduate-research-writing-studies/.
Klos, N., Shanahan, J., & Young, G. (2011). Creative inquiry in the arts & humanities: Models of undergraduate research. Washington, DC: Council on Undergraduate Research.
Kuh, G. (2008). High-Impact educational practices: What they are, who has access to them, and why they matter. Washington, DC: Association of American Colleges and Universities.
McKee, H., & DeVoss, D. (Eds.). (2007). Digital writing research: Technologies, methodologies and ethical issues. New York: Hampton Press.
Moore, J. (2017). Review of researching writing: An introduction to research methods. Journal of Second Language Writing, 38, 1–3.
Pappas, C., & Tucker-Raymond, E. (2011). Becoming a teacher researcher in literacy teaching and learning: Strategies and tools for the inquiry process. New York: Routledge.
Perdue, S., Driscoll, D., Matthews, J., Paz, E., & Tess, J. (2014). Negotiating the sponsorship continuum: Preparing humanities undergraduates to conduct RAD research. PURM: Perspectives on Undergraduate Research and Mentoring, 3(2), 1–19.
Ray, R. (1993). The Practice of theory: Teacher research in composition. Urbana, IL: National Council of Teachers of English.
Saldaña, J. (2015). The coding manual for qualitative researchers (3rd ed.). SAGE Publications.
Saldaña, J., & Omasta, M. (2018). Qualitative research: Analyzing life. SAGE Publications.
Scott, J., & Meloncon, L. (2017). Writing and rhetoric majors, disciplinarity, and techne. Composition Forum. https://compositionforum.com/issue/35/majors.php.
The Council on Undergraduate Research. (2017). Code of ethics for undergraduate research. https://www.cur.org/assets/1/7/Code_of_Ethics.pdf.
Traywick, D. (2010). Preaching what we practice: RCR instruction for undergraduate researchers in writing studies. In L. Grobman & J. Kinkead (Eds.), Undergraduate research in English studies (pp. 51–73).
Tufte, E. (2001). The visual display of quantitative information (2nd ed.). Graphics Press.
U.S. Department of Health & Human Services Office for Human Research Protections. (2018). Protection of human subjects, 46.102. Definitions for purposes of this policy. https://www.ecfr.gov/cgi-bin/retrieveECFR?gp=&SID=83cd09e1c0f5c6937cd9d7513160fc3f&pitd=20180719&n=pt45.1.46&r=PART&ty=HTML#se45.1.46_1102.
Wong, D. (2013). The Wall Street Journal guide to information graphics: The dos and don’ts of presenting data, facts, and figures. New York: Norton.
See Pappas and Tucker-Raymond (2011) and Ray (1993) for teacher research.↩
Excellent resources on coding include Saldaña (2015), Saldaña and Omasta (2018) and Grant-Davie (1992)↩
See Fitzgerald and Midiri (2013) on the importance of mentored research.↩
A few students generally were already certified and did not have to complete this assignment. They automatically get the points designated for certification.↩
For additional reading on standards of ethics when doing research, particularly in English Studies, see Anderson et. al (2001); The Council on Undergraduate Research (2017); Traywick (2010). Additional ethical complexities and responsibilities come to the fore when students wish to study social media (see McKee & DeVoss, 2007). What may seem like public writing on sites such as Facebook, Instagram, or Snapchat requires informed consent from participants.↩
Human subjects is the term used by the federal department overseeing this type of research. Many researchers prefer the term human participants and have found ways for people not only to be studied but also to participate in meaningful ways in the findings.↩
IRBs vary by campus. I am fortunate that IRB staff members provide training to faculty and students, include templates for forms online, and move through applications fairly quickly. The individual projects undertaken by my students resulted in some policy and practice changes for our course-based research process, and I was also invited to join the IRB. I view these as positive developments.↩
SurveyMonkey and Qualtrics both offer tips on asking survey questions. See https://www.surveymonkey.com/mp/writing-survey-questions/ or https://www.qualtrics.com/blog/good-survey-questions/.↩
Our campus has a subscription to Qualtrics through our Research Office.↩
The USU IRB has a page of resources that includes informed consent templates and advice for recruitment scripts, as well as an Investigator’s Handbook (http://rgs.usu.edu/irb/resources/).↩
Our campus dairy store offers grants for educational and research projects; we were fortunate to receive one of the monthly awards that subsidized the purchase of ice cream.↩
An excellent overview to a specific undergraduate research project is Janangelo’s case study, “Anatomy of an Article” (2010) that focuses on one student, the study, and the path to publication in YSW.↩
See Tufte (2001) and Wong (2013) for advice on graphic depiction of information.↩
Note that data is a plural word and requires a plural verb for correct subject-verb agreement.↩
The Informed Consent Letter requires a signature; it was used for in-person faculty interviews but contains much of the same verbiage as the Letter of Information.↩