Author Podcast (2010): Ryan Boettger on This Paper

In 2010, IEEE Transactions on Professional Communication invited authors to record a short podcast about their article — part of editor-in-chief Jo Mackiewicz’s effort to promote the journal. The audio is long gone, but here is the script I recorded: a snapshot of how I described this work at the very start of my career.

Hi. This is Ryan Boettger. I’m with the University of North Texas in Denton, and this is a podcast for my article, “Rubric Use in Technical Communication: Exploring the Process of Creating Valid and Reliable Assessment Tools.”

My article appears in the March 2010 issue of IEEE Transactions on Professional Communication, which is the peer-reviewed journal of the IEEE Professional Communication Society. This article is also part of a special issue on assessment in professional communication.

I’m going to briefly describe the contents of this research, but I would also like to offer some background on what inspired me to write this article, and how I see it fitting in with future research in technical and professional communication.

But first, let me define rubric and give you an overview of the different types of rubrics used for assessment. I define a rubric as a criteria-based scoring guide that provides some measurable assistance for evaluating a product or service, if you’re using one in a workplace context, or the quality of student writing, if you’re using one in a classroom environment.

You’ve probably all seen a rubric, even if you’re unfamiliar with the term. Rubrics are always designed with a number of achievement scales, and these scales help the evaluator consistently assess the performance. These scales can be labeled with numbers, word descriptors, or both. For example, a student who just completed the essay section of the GRE might score a 5 for that essay, which represents a “superior” performance, or, to illustrate the other end of the spectrum; they may receive a 1, which represents a “failing” performance.

There are also two types of rubrics: holistic and analytic. Deciding which rubric form to use will depend primarily on the assessment purpose. A holistic rubric captures an impression of overall quality, which is then represented by a single numeric score. For example, if a manager wanted to assess her employees’ overall writing proficiencies on a quarterly basis, she would use a holistic rubric because it offers a snapshot of overall quality. The manager could eventually use this data to track her employees’ writing proficiencies longitudinally and these results could possibly inform decisions related to promotions or revised hiring practices.

The second type of rubric is the analytic form. An analytic rubric assesses a performances on a series of mutually discreet traits, including content areas, writing style, and document design. Each trait receives a single numeric score that can then be averaged into an overall assessment. This approach is recommended for day-to-day use because it offers writers guided feedback for revision.

So let me go back to my earlier example of the manager needing to assess her employees’ writing. Let’s say that these employees have had issues with style in their correspondence with clients, and the manager wants to assess her employees on three defined traits: the tone the writer establishes to convey information to clients, the writers’ use of word choice, and the writers’ level of formality. Evaluating writing on these three traits is going to provide the manager with much more information than a holistic assessment. If the manager is conducting this assessment to help her employees refine their communication style, employees will appreciate having specific information on what traits they need to improve on versus the traits they may have already mastered.

So the first part of my article is focused on demonstrating how assessment rubrics have been used in the field of professional and technical communication, and how new tools can be created to assess some of the questions the field has raised. In the academic discipline of technical communication, rubrics are primarily used in the classroom and for accreditation purposes, but several scholars have called for assessment tools sophisticated enough for comprehensive statistical analyses, particularly in experimental and ethnographic studies. One of the core arguments in my article is that rubrics are perfect tools for these research purposes.

Likewise, the first part of my article discusses ways that rubrics can be constructed to ensure validity and reliability. As I discuss in the article, the limited rubric research in technical communication scholarship offers a variety of ways to ensure the validity and reliability of the assessment process. Some of these approaches are more successful than others, and so I also pull information from psychometricians and other disciplines like Teaching English to Speakers of Other Languages in an attempt to provide researchers and practitioners with a discussion of agreed-upon assessment standards.

The second part of my article illustrates all this assessment theory and agreed-upon best practices with a case study that documents the development and implementation of one holistic and five analytic rubrics used to evaluate undergraduate projects in technical communication.

I created these rubrics to assess the effects of an explicit teaching pedagogy versus a more traditional, lecture-based approach to teaching. Explicit teaching is defined as any discussion of a genre’s formal features, including the discussion of the cultural, political, or social factors that shape these features. It was hypothesized that students learning via explicit teaching would produce documents that targeted audience and purpose more successfully than the students learning via the more traditional methods.

The case study describes the process of revising an existing holistic rubric that assessed the undergraduate students’ writing proficiencies and creating five site-specific analytic rubrics that each assessed a specific technical communication genre: the resume, the job letter, the claim letter, the recruitment email, and the instruction set. The case study also details how I trained four student raters on the rubrics to maintain a statistically significant level of agreement.

As a teacher of technical communication and as a pedagogy researcher, I am interested in exploring ways to best teach my students, and in turn, empirically assess the effectiveness of those pedagogical practices. So when the call for proposals for this special issue came out, I saw it as a perfect forum for presenting, essentially, a composite of my researcher’s notebook that documented my observations in creating the assessment rubrics as well as the successes and sometimes pitfalls involved in training the four student raters.

For me, the case study makes an important contribution to professional and technical communication. First, to my knowledge, no other study in the field has devoted so much discussion to the design of an actual assessment process, a description of what validity and reliability issues arose, and how those issues were addressed. I think narratives like this are becoming increasingly important for our field in particular because our country’s education system has become increasingly “assessment sensitive” due in part to the No Child Left Behind Act of 2001, which requires teachers and program directors to quantify their own learning outcomes and to collect and analyze data on their students’ achievements.

The first full generation of NCLB learners will enter the college classroom as early as 2014 and they will arguably be most accustomed to the structured assessment that rubrics provide. In my opinion, technical communication teachers carry the brunt of the pedagogical and practical assessment burdens because we’re charged with preparing our classroom students to become valuable workplace communicators. We need to continue exploring ways that best assess our students’ progress but can also prepare them to become productive workplace employees. The continued, mindful exploration of future assessment practices, like rubrics, will enhance the field’s knowledge and increase its arsenal of best practices.

I hope you take something away from my article. Please also visit the IEEE digital library because all six of rubrics that I created for my study are housed there. I hope you can find some use for these rubrics in your own research, but please remember that I created these assessment tools for my own site-specifics purposes. Make sure you adapt what I’ve done for your own purposes and your own program. Please feel free to send me your feedback via email at ryan dot boettger at unt dot edu. And thank you for listening.