Abstract: The Unbibliography asks students to keep track of sources they thought they might use in an annotated bibliography assignment but ultimately rejected. Each discarded source is annotated with details about these two moments—when the source seems valuable and when it proves less useful than it originally promised—in the research process. The project also includes a component that requires students to reflect on how the Unbibliography impacted their experience of developing the annotated bibliography project. By highlighting and valuing a part of the research process that is typically regarded as a waste of time, the Unbibliography resituates “failure”—identifying and discarding potential sources—as an essential part of the process. In this way, students are encouraged to grow from novice to experienced researchers.

Introduction

Research—an integral part of writing in an academic context (Bodemer 2012; Elmborg 2005; Isbell and Broaddus 1995)—is often described as a process of trial and error. This characterization has the potential to help students think of research as iterative, non-linear, and messy, yet most documents that students are asked to produce for their classes focus on demonstrating success without providing an opportunity to reflect on how false starts, missteps, and failure can be valuable parts of the research journey. For example, an annotated bibliography project prompt might include information on how many sources a student should include without providing any guidance on how many sources should be consulted along the way. To the novice researcher, this may imply that if the assignment requires five sources, the student or researcher can just use the first five they come across that meet the basic criteria. Experienced researchers know the satisfaction of developing a complex project based on a wide range of sources that may or may not have made the final cut, but novices often need scaffolded help to come to the conclusion that a grab-and-go approach to research is insufficient. The Unbibliography assignment is designed to help them move towards that conclusion.

I have most recently assigned the Unbibliography to my own students as a graduate instructor of record in the First Year Writing (FYW) program of the University of Connecticut. I first articulated it, however, as a suggestion to a writing department faculty partner while working as an instruction and reference librarian at George Washington University. The Unbibliography is rooted in an idea I encountered while studying to be a librarian called the Rule of Three: a student should examine three times as many sources as called for in the assignment prompt. When working on an annotated bibliography that asks for five sources, then, the student will produce the best work if they examine at least fifteen sources rather than settling for the first five they encounter. Thus, the Unbibliography asks students to keep track of sources they thought they might use in the annotated bibliography assignment but ultimately rejected. Each discarded source is annotated with details about these two “decision points” (Elmborg 2005, 11): the initial rationale for choosing the source and the explanation for discarding it. The project also includes a component that requires students to reflect on how the Unbibliography impacted their experience of developing the annotated bibliography project. By highlighting and valuing a part of the research process that is typically regarded as a waste of time, the Unbibliography resituates “failure”—discarding sources that initially looked promising—as an essential part of the process. In this way, students are encouraged to grow from novice to experienced researchers.

Of course, the Unbibliography is not the first attempt to adapt or extend annotated bibliography assignments. For example, Jacob D. Richter ((2020)) assigns an Infosphere Probe which expands the media streams which are acceptable and even necessary to explore and include. Allison Hosier ((2015)) asks her students to write an “un-research” (p. 126) essay based on knowledge they already possess; students then search for sources that support, challenge, enhance, or “add new information” (p. 130) to their essays. The Unbibliography differs from these reimaginings of the annotated bibliography assignment by shifting the focus away from the finished product and toward the cutting room floor.

Information Literacy and the Unbibliography

Annotated bibliographies are often assigned to “jump-start the research process in preparation for a larger assignment or project” (Richter, (2020), p. 26; see also Mantler, (2017)). My experience as an instruction and reference librarian, however, has taught me that the task of listing and describing adequate sources can only go so far in prodding students to grow as researchers. One of the information literacy concepts articulated by the Association of College and Research Libraries (ACRL) ((2015)) in their Framework for Information Literacy for Higher Education is “searching as strategic exploration” (p. 22). They explain that “novice learners may search a limited set of resources, while experts may search more broadly and deeply to determine the most appropriate information within the project scope” (p. 22). In other words, an emerging sense of information literacy involves becoming accustomed to looking at more sources than students might use for a particular project. This aspect of information literacy makes the Rule of Three particularly relevant. Most students, however, are either unaware of this advice or unfamiliar with the benefits of following it. Warwick et al. (2009) found that students tended to engage in what they call “strategic satisficing” (p. 2409). In other words, students in their study would “create time-saving strategies to complete the coursework with minimum effort rather than harnessing their skills to complete an excellent assignment” (p. 2409). I do not fault students for seeking to manage their limited time wisely. Instead, I use the Unbibliography to incentivize them to “search more broadly and deeply” than they might ordinarily do.

The Development of the Unbibliography

One concept that frequently arose when I collaborated with writing department faculty as a librarian was helping students evaluate sources. My approach to this skill has been to ask students to bring in sources they knew they would not or could not use for the assignment. Our conversation about why they would or could not use those sources enabled the class to develop criteria to guide their research process. These criteria typically matched those articulated by the CRAAP test (Currency, Relevance, Authority, Accuracy, Purpose; see Blakeslee, (2004)); when there were gaps I would offer suggestions. Understanding how to evaluate sources and examining enough sources are, however, two different skills. Based on the Rule of Three, I suggested to a few of my faculty partners that they require students to submit a list of sources the students had looked at but decided against using. One of my faculty partners liked the idea and assigned the project.

At first, students would often report that they rejected a source because it was not on their topic. While relevance is an important evaluation criteria, the discarded sources were often wildly off topic which gave the impression that students may have thrown together a list that had little or nothing to do with the process of finding sources they actually intended to use. In other words, these students had not been adequately discouraged from strategic satisficing. To address this problem, I worked with the faculty member to adjust the prompt in following semesters to ask students to examine sources, not just look at them. The faculty member and I also clarified what was meant by “examine,” namely that the Unbibliography should include sources the student initially thought they might use but decided against. The annotations written in response to the tweaked prompt revealed that the adjustment was successful: students still discarded sources based on relevance but did so in a more nuanced way.

As both a librarian and an instructor of record, I have worked with annotated bibliography projects and the Unbibliography as intertwined assignments. For that reason, I have made sure to reference the latter when teaching the former. For example, I typically use topics volunteered by students to drive our exploration of library databases. This has meant that the student in question could guide our search process by indicating whether the first few sources on a results list seemed relevant or acceptable. Before demonstrating how to refine the search to get better results, I would remind the students that whatever source prompted the need to refine was fair game for the Unbibliography. I modeled the Unbibliography annotations with statements like “I thought I might use this article because the title is spot on for my topic but on further examination, I decided it was too old.” It was my hope in doing this that the Unbibliography would remain at the forefront of students’ minds and that it would feel possible.

The Impacts of the Unbibliography

In the fall of 2022, I included the Unbibliography as an optional assignment for a television-themed FYW course I taught at the University of Connecticut. In the context of a labor-based grading contract, an annotated bibliography was required of all students while the Unbibliography was positioned as one of many ways a student could pull their grade up from a B towards an A. Most of the optional assignments for this class had the end of the semester as a due date, but I stipulated that the Unbibliography had to be turned in at the same time as the required annotated bibliography to emphasize the fact that these assignments were meant to operate in tandem.

In preparing to write this essay, I solicited and received written permission from seven students to anonymously refer to and quote from their projects. Because the writing was reflective in nature, I was able to get a good sense of students’ experiences of doing the Unbibliography. I believe this assignment was a success for these students; some said outright that the Unbibliography made them a better researcher or writer. I got a sense that I had converted more than one to the Rule of Three strategy of research because their comments indicated that the project had changed how they planned to conduct research in the future. More than one student reported that before doing this assignment, they hadn’t realized how helpful it would be to have a variety of sources to choose from.

Students who completed the Unbibliography assignment showed evidence of leaving strategic satisficing behind as a method of doing research. One student reported that “this was…a good project to do because it proved to me that you have to comb through so many sources to find usable ones. I could not just pick the first five I came across and expect my annotated bibliography to make sense for my research question.” In other words, without the Unbibliography, this student would likely have stopped examining sources once she found five that met the criteria for the annotated bibliography. Instead, she read “over thirty sources to find just five to use for [her] project,” even though she only needed to look at fifteen to satisfy the requirements of the combined assignment. Another student echoed these observations, explaining that he compiled a “long list of possible sources” so that he wouldn’t have to search again for sources for the Unbibliography. He added: “I think doing this helped me pick better sources for my annotated bibliography because I had so many to choose from.” This suggests that this student may choose to recreate this research process for future projects even if looking at more sources than needed is not required or explicitly rewarded.

Part of the assignment requires students to provide a specific reason for rejecting each source listed on the Unbibliography. Sometimes students listed quality issues as reasons to not use a source—it was too old, or it was “just a list of facts.” More often, though, the reasons had to do with relevance of the source to the student’s research question. For example, one student explained that “since my research subject was exclusive to the United States and this piece was for worldwide television, I decided not to utilize it.” One trend in the annotations that I was not expecting was that a number of students reported listing a source on the Unbibliography rather than the annotated bibliography because they had found another source that was more successful at meeting their needs. For example, a student wrote that “other articles discussed the same issue with more excellent knowledge, prompting me to utilize something other than this.” The source in question was not flawed, it was simply not the best amongst what the student had found. Many students may not have come to this conclusion without having been encouraged to and rewarded for extending their research beyond the first sources that met the criteria.

Another unexpected but related trend in the student reflections was that the Unbibliography helped more than one student realize that relevance (like quality) is not a static concept, but instead is fluid and dynamic. What may have seemed relevant at the beginning of the research process may have been entirely irrelevant by the end. The shifting nature of the relevance of a source was most obvious when a student changed their topic entirely, but sources also became less suitable as students refined their research questions. At times, the very process of reviewing and rejecting a particular source led the student to revise their research question. For example, one student started with the research question “Do actors have more engagement within television when they are active in media elsewhere?” In one of his annotations, he indicated that “this article made me rethink how I would think about actor engagement;” his final research question was “How does an actor’s activity within social media affect the viewer’s engagement on television in the United States?” The new research question was more specific and ultimately more generative.

Another student reported that her final research question was shaped by trends in the research she was finding: “Many of the other sources I was using focused on the reading, writing, and speech development of younger children. This source was not in alignment with those other parameters.” The source in question was relevant to her original, broad research question, but as that research question was refined in response to the available scholarship, the idea of what was relevant to the project changed. I was very pleased to see this trend in the students’ reflections as it demonstrated that students were beginning to understand that “searching for information is often nonlinear and iterative, requiring the evaluation of a range of information sources and the mental flexibility to pursue alternate avenues as new understanding develops” (Association of College and Research Libraries 2015, 22). The students’ mental flexibility was stretched and expanded as they evaluated the information sources in front of them and they had more sources in front of them than they would have had without the Unbibliography assignment. These students have grown into sophisticated researchers who value so-called failures as necessary steps in the process of developing their future research projects.

Successes, Limitations, and Ideas for the Future

In my experience, students often approach research as though it was called “refind”—repeating the process of finding acceptable sources rather than exploring the available sources until they have the best. The Unbibliography allows students to both practice the Rule of Three and reflect on how deploying this strategy helps them craft better research projects. One limitation of the Unbibliography as I have assigned it, however, is that all students would benefit from the Rule of Three but not all students engaged with it. More students indicated they were going to do the Unbibliography than actually turned it in. This suggests to me that some students found the process to be more difficult or more time-consuming than they had anticipated. One who turned it in said he thought it would be easy but was proven wrong. An adjustment I may make in the future is to require the Unbibliography of all students rather than making it an opt-in assignment. Unfortunately, another limitation of the assignment is that it may not scale up very well. In the fall of 2022, I assessed five citations and annotations from thirty-two students in addition to ten each from the nine students who completed the Unbibliography—a total of two hundred fifty. If all thirty-two students turned in the combined project, however, I would need to assess four hundred eighty citations and annotations. The advantages for the students may or may not outweigh the consequences for my time and for the other projects and priorities of the syllabus.

As the Unbibliography continues to evolve, I intend to lean into the metacognitive underpinnings of information literacy as I update the framing of the assignment for my students. The ACRL defines metacognition as “an awareness of one’s own thought process” (Association of College and Research Libraries 2015, 9) and positions its development as an essential part of moving from novice to expert researcher. The expanded prompt included as supplementary material is one outcome of my increased commitment to foster metacognitive growth. I anticipate that having a written account of the purpose and benefits of the Unbibliography will not only help students asynchronously but also spark in-class conversation about the hows and whys of the assignment. These conversations will encourage students to consider how they think about research and how that conception of the research process might expand by opting into the Unbibliography assignment.

Finally, a possible extension of the Unbibliography would be to pair it with a research project other than an annotated bibliography. Because “scholarship is a conversation” (Association of College and Research Libraries 2015, 20), any research project involves locating, assessing, and engaging with past research. Students often must also find, evaluate, and choose quotes, data sets, or other objects of study. Either circumstance presents an opportunity to introduce and incentivize the Rule of Three. For example, a sociology instructor might pair a twenty-source Unbibliography with a ten-source literature review. Alternately, a literature instructor might ask students to find and reflect on nine quotes from a text they are meant to close read before choosing the three that best help make their argument. The Unbibliography would be a valuable resource to any instructor who wanted to scaffold the research process and make it more transparent regardless of the shape taken by the target project. In adopting this kind of assignment, instructors can guide students into an appreciation of the value of the messy parts of the research process that might otherwise feel like a waste of time. By reframing apparent failure as an integral part of the process, instructors can invite students into a more sophisticated relationship with the concept of research.

ASSIGNMENT
Unbibliography

Unbibliography (4 points¹): A list of ten sources that you considered for your Annotated Bibliography but ultimately rejected with annotations indicating why you considered each item and why you decided not to use it. At least three of these rejected sources must be scholarly. You will also write 1-2 pages on how this assignment shaped your thinking about your Annotated Bibliography. This assignment must be turned in with your Annotated Bibliography.

[Editor note: An expanded version of the original assignment the author intends for future use is included as a supplement to this article (see Supplementary Materials.]

Notes