LOEX: Using Benchmarks to Measure Library Instruction Progress and Success

Candice Benjes-Small and Eric Ackermann from Radford University spoke about how they redesigned their assessment process for instruction. They had reached a point where merely counting number of sessions was deemed no longer useful in measuring success.

All librarians had been using a standard student evaluation form that had a four point Likert scale and a single comment box at the end. They found the disconnect between the scores and the comments to be problematic and not useful in making changes. It was decided to modify the evaluation form to ask for qualitative feedback for each question.

The modified evaluation form asks the following three questions

1. I learned something useful for this workshop.

  • Strongly Agree: Name one thing you learned from this workshop?
  • Agree: Name one thing you learned from this workshop?
  • Disagree: How can the workshop be improved?
  • Strongly Disagree: How can the workshop be improved? 

2. I think this librarian was a good teacher.

  • Strongly Agree: What did you like about the teaching?
  • Agree: What did you like about the teaching?
  • Disagree: What did you dislike about the teaching?
  • Strongly Disagree: What did you dislike about the teaching? 

3. I would recommend this workshop to someone interested in library research.

  • Strongly Agree
  • Agree
  • Disagree
  • Strongly Disagree 

They chose a comment based metric methodology for assessment. This is similar to what the University of Virginia Library is doing with their balanced scorecard metrics.  "What did you dislike about the teaching?" was chosen as the question to measure. This would allow for the librarian teaching to have something tangible for improving instructional delivery. A target of less than 5% negative comments was set to be the measure for total success. Partial success would be achieved if 5 to 10% of the comments were negative. 

Advantages

  • Evidence based
  • Allows for goals to be set and measured
  • Flexible to measure what you want to know

Disadvantages

  • Time intensive, especially coding qualitative comments
  • Difficult to change evaluation forms if you want to go back and measure another goal 

Questions to consider

  • What do you want to know?
  • How are you going to measure? 
  • Are you going to focus on evaluation scores (quantitative) or comments (qualitative)?
  • What is the target for success?
  • Who is going to compile the results?

Their PowerPoint slides are available.

Dual Assignments Discussion Group

I was about 15 minutes late arriving at the Dual Assignments Discussion Group. Thirteen people had found McPherson A and B tucked away at the end of the third floor of the Four Points Shearton. Most of the participants were from academic libraries based on the sign-in sheet. 

The publicized topic is what drew me to this discussion group, "Collection Assessment: Analysis and Decision".

Libraries of all kinds are assessing collections using tools such as WorldCat Collection Analysis and their integrated library systems.  The results lead to important decisions about what to add, what to preserve, what to discard, and what to replace. And how do we explain to concerned users why these choices were made?

I'm interested in this topic for two reasons. First, our circulating collection has not been evaluated, assessed, pondered, let alone weeded in a very long time. Second, I'm on OhioLINK's Collection Building Task Force and one of our current projects is a statewide collection assessment.

My impression from the conversation is that everyone in the room was very interested in collection assessment and some had started to do projects with the various tools. It seemed that many were either in the planning phase or in the data analysis phase. Most had not gotten to the decision/action phase.

Lack of space was mentioned a couple of times as a driving factor for collection assessment. Planning to commit to electronic only versions of resources was given as another reason. The conversation ranged from dark storage to electronic reference books to disposition of withdrawn print materials.

I like discussion groups, because I get many good ideas that can be tried at my own library. One of the take-aways for me was hearing other libraries' experiences with Better World Books. My colleague had talked to them at the ACRL conference and recommended that we try it out. It was good to hear positive comments from other participants. One attendee cautioned the group to make sure it is okay locally to dispose of withdrawn items this way. Some state universities have restrictions on how they can dispose of items purchased with tax payer money.

Another lesson I learned from this session is to take better notes. I wrote down…metrics: look at percentage of circulation, percentage of the collection, and percentage of the materials budget.  Sounds good. Now if only I could remember the context.

Towards the end of the discussion Allison Cowgill suggested that this topic be brought up again in a year or two. She hoped that by that time, participants would be able to share more about how they communicating actions taken based on assessment with faculty and students.

I OCRed the bibliography that Jessica and Allison distributed. It's not on the RSS or CODES site yet and I'm not sure it will be.

Dual Assignments Discussion Group
Reference and User Services Association
Reference Services Section and Collection Development and Evaluation Section
June 23, 2007

Collection Assessment: Analysis and Decision 

Agee, Jim. "Collection Evaluation: A Foundation for Collection Development." Collection Building 24, no. 3 (2005): 92-95.

Baird, Brian J. Library Collection Assessment Through Statistical Sampling. Lanham, MD: Scarecrow Press, 2004. 103 pp.

Ballestro, John, and Philip C. Howze. "When a Gift Is Not a Gift: Assessment Using Cost-Benefit Analysis." Collection Management 30, no. 3 (2005): 49-66.

Brewer, Michael. "Identifing Holdings Unique to Your Library’s Collections Using WorldCat." Slavic & East European Information Resources 7, no. 4 (2007):115-121.

Bushing, Mary, Bums Davis, and Nancy Powell. Using the Conspectus Method: A Collection Assessment Handbook. Lacey, WA: WLN, 1997. 200 pp

Dilevko, Juris, and Lisa Gottlieb. "Weed to Achieve: A Fundamental Part of the Public Library Mission?" Library Collections, Acquisitions & Techical Services 27 (Spring 2003): 73-96.

Hiott, Judith, and Carla Beasley. "Electronic Collection Management: Completing the Cycle – Experiences at Two Libraries." Acquisitions Librarian 17, no. 33/34 (2005): 159-178.

Hoffman, Frank W. Library Collection Development Policies: Academic, Public, And Special Libraries. Lanham, MD: Scarcrow Press, 2005. 329 pp.

Johnson, Peggy. Fundamentals of Collection Development & Management. Chicago, IL: American Library Association, 2004. 342 pp.

Knievel, Jennifer E., Heather Wicht, and Lynn Silipigni Connaway. "Use of Circulation Statistics and Interlibrary Loan Data in Collection Management." College & Research Libraries 67 (January 2006): 35-49.

Lesniaski, David. "Evaluating Collections: A Discussion and Extension of Brief Tests of Collection Strength." College and Undergraduate Libraries 11, no. 1(2004): 11-25.

Lyons, Lucy E. "A Critical Examination of the Assessment Analysis Capabilities of OCLC ACAS." Journal of Academic Librarianship 31 (November 2005): 505-516.

Metz, Paul, and Caryl Gray. "Perspectives on: Public Relations and Library Weeding." Journal of Academic Librarianship 31 (May 2005): 273-279.

Mitchell, Steve. "Machine Assistance in Collection Building: New Tools, Research, Issues, and Reflections." Information Technology & Libraries 25 (December 2006): 190-216.

Mortimore, Jeffrey M. "Access-informed Collection Development and the Academic Library: Using Holdings, Circulation, and ILL Data to Development Prescient Collections." Collection Management 30, no. 3 (2005): 2 1-37.

Nisonger, Thomas E. Evaluation of Library Collections, Access & Electronic Resources:
A Literature Guide & Annotated Bibliography
. Westport, CN: Libraries Unlimited, 2003. 316 pp.

Oberlander, Cyril, and Dan Streeter. "LibStatCAT: A Library Statistical Collection Assessment Toll for Individual Libraries & Cooperative Collection Development." Library Collections, Acquisitions & Technical Services 27 (Winter 2003): 493-506.

Perrault, Anna H., Tina M. Adams, Rhonda Smith, and Jeannie Dixon. "The Florida Community College Statewide Collection Assessment Project: Outcomes and Impact." College and Research Libraries 63 (May 2002): 240-249.

Shouse, Daniel L., and Linda Teel. "Inventory: Catalyst for Collection Development." Collection Building 25, no. 4 (2006): 129-133.

Slote, Stanley J. Weeding Library Collections: Library Weeding Methods. 4th ed Englewood, CO: Libraries Unlimited, 1997. 240 pp.

Stoller, Michael. "Building Library Collections: It’s Still About the User." Collection Building 24, no. 1 (2005): 4-8.

Teper, Thomas H., and Stephanie S. Atkins. "Building Preservation: The University Of Illinois at Urbana-Champaign’s Stacks Assessment." College & Research Libraries News 64 (May 2003): 211-227.

Jessica Moyer, RSS
Allison Cowgill, CODES
Co-Chairs
June 20, 2007 

technorati tag: Annual2007

ALAO 2006: Gourmet Cooking on a Budget? A Frugal Recipe for Library-wide Assessment

Colleen Boof, Catherine Cardwell, and Kelly Broughton gave this presentation on BGSU's assessment plan. They distributed a document showing their success measures and priorities pulled from their complete assessment plan . The remainder of the session was a group discussion on how others are assessing their library and services.