Organization Monkey

This post is part of a series, describing the assessments used to develop the Institute for Research Design in Librarianship (IRDL).

At the end of the program we checked in with the Scholars via a questionnaire, to learn at what point they were in completing their research projects. We asked if they had begun any other new research, as well as anything else research-related in the past year. In an open-text response, we asked the Scholars what surprised them most about their IRDL year, if they experienced unexpected advantages or disadvantages from their participation in the program, and if they had any specific changes to suggest to us about the pre-summer-workshop activities, the workshop itself, or support structures after the workshop.

Regardless of what stage of completion of their project the Scholars were in, overall they were surprised to feel so motivated and “well-equipped” to complete their projects. They commented throughout the open-ended text responses about the necessity of the ongoing peer support and mentor support.

In response to the question about what they were most surprised about, one Scholar commented, “I’m surprised at how confident I feel. Not that I feel over confident, but I don’t feel scared or hesitant to give something new a try with regards to research.” Another noted, that, “I did not anticipate the critical importance and wonderful opportunities to develop a valuable research network, which have been amazing.”

Linked here is the version of the survey in use from 2022-2024.

My reflection on the use of this tool for assessing the program
The results of this anonymous survey were affirming for many reasons. A main goal of the program was to increase the number of librarians publishing quality research, so it was good to see that within a year the Scholars had completed many of the pieces of their research project, with some even having published their results. Their responses demonstrate that the program model is working as intended, to give them the tools and support needed to succeed in conducting novel research.

The cost of this assessment tool
Because LMU has an institutional subscription to Qualtrics, no grant funds needed to be used for the assessment.

Introduction post, Confidence scale, Research networks of the Scholars, External review, Post-workshop survey, Pre-/post-workshop research proposal evaluation, Mid-point check-in survey

This post is part of a series, describing the assessments used to develop the Institute for Research Design in Librarianship (IRDL).

Five months after the Summer Research Workshop was complete, we emailed them a link to a Qualtrics survey that took about five minutes to complete, to ask about their workshop experience. The survey was not anonymous. LMU’s IRB reviewed the protocol ahead of sending it to the group.

I wanted to measure the component of Bandura’s theory of self-efficacy, emotional arousal (which includes habits and stressors). We also wanted to understand institutional/organizational support/culture that is in place for the Scholars during their IRDL year. To invite further communication about any challenges the Scholars were experiencing post-workshop, the last question in the survey was, “Would you like us to reach out to you to schedule a Zoom check-in with Marie and Kris?”

In the questionnaire we asked the Scholars to rate their general level of agreement on a number of issues related to time and scheduling, space and environmental factors, competing priorities, affect, and social support, to learn how personal habits, stressors, and institutional support may influence the issues. I used a Likert scale of response options: 1, strongly agree, 2 somewhat disagree, 3 somewhat agree, 4 strongly agree. Some example questions from the survey:

1. I am able to work four hours or more per week on my IRDL project while I’m “on the clock”
2. I’m able to get “in the zone” and concentrate uninterrupted when I’m working on my IRDL project
3. I’ve been able to prioritize working on my IRDL project even though there are work tasks competing for my time
4. There’s stuff going on in my personal life that is overwhelming the time I need to work on my IRDL project
5. My supervisor is supportive of me working on my IRDL project

Linked here is the version of the survey in use from 2022-2024.

My reflection on the use of this tool for assessing the program
The results of the survey helped us better appreciate the environment in which the Scholars were conducting their research projects. We were then better poised to provide options for pivots in research design, extended timelines, or pauses in completion of their projects, to model a positive work-life balance. Our hope for the Scholars was that they would be able to insert conducting research into their regular lives and having examples of what the challenges they were experiencing helped us center that in the model we developed.

The cost of this assessment tool
Because LMU has an institutional subscription to Qualtrics, no grant funds needed to be used for the assessment.

Earlier posts in this series:
Introduction post, Confidence scale, Research networks of the Scholars, External review, Post-workshop survey, Pre-/post-workshop research proposal evaluation

This post is part of a series, describing the assessments used to develop the Institute for Research Design in Librarianship (IRDL).

Participants submitted a research proposal with their applications and then worked on revising the proposals during and after the Summer Research Workshop ended. Once the revised proposals were submitted one month after the end of the workshop, we conducted a review process to compare the pre-workshop proposals (the proposal submitted as part of the application) with the post-workshop proposals (the proposal that is revised throughout the workshop) on multiple criteria.

We developed the evaluation process to help us learn if there was a change in how the Scholars wrote about their research plans, from before the workshop to after. To measure any change, we created a rubric that guided us to look in the proposals for mastery of specific components of their research plans. One of the components, for example, was to look at the proposal and consider the scope, if the design they selected was appropriate for their project’s needs. We wanted the Scholar to design a project that was reasonable for a novice researcher to complete, given their needs for information, resources, and time. On the rubric for that component, a Scholar would score a 1 (Beginning) if we saw evidence in the proposal that there were few components of the proposal that can be feasibly completed, based on the information needed, resources needed, and timeline identified. A Scholar would score a 2 (Developing) if some of the components can be feasibly completed, based on the information needed, resources needed, and timeline identified. A Scholar would score a 3 (Accomplished) if many of the components of the proposal can be feasibly completed, based on the information needed, resources needed, and timeline identified. A Scholar would score a 4 (Exemplary) if the majority of the components of the proposal can be feasibly completed, based on the information needed, resources needed, and timeline identified.

During the workshop, in group discussions and one-on-one consultations with experts, we worked with each Scholar to design their projects so that they would have success within the one-year program time frame. This often meant treating a project as a phased project, so they fully completed a part of the project within the one year, before moving on to the next phase. If a Scholar was committed to completing their full project in one year, we discussed alternatives for the amount or type of data that could be gathered or prompted them to seek help from a colleague to share the work. Our expectation for each Scholar was that they would leave the workshop with a project that was in scope for their individual work-life situation, so they would be more likely to be successful in completing it. We expected to see in their revised proposal specifics about the steps they would take within the year, with the various phases of the project mapped to a timeline.

Each of the components measured on the rubric were specifically addressed during the workshop. The other components were: significance and purpose, wanting the Scholar to demonstrate a consideration of the potential impact of their research; research question/hypothesis, wanting the Scholar to produce a project that relies on a well-constructed research question or hypothesis; literature review organization, wanting the Scholar to construct a clear review of the literature; literature review explanation, wanting the Scholar to make effective use of the literature to support the thesis in their proposed project; methods – research design, wanting the Scholar to select methods relevant to their problem statement; methods – context, population, and sampling, wanting the Scholar to select appropriate members of the population to include in the project; and methods – procedures, wanting the Scholar to consider the permissions, treatments, and data gathering. All of these components were measured on a 1 to 4 scale, with a rationale for each.

To prepare the proposals for evaluation, the office of the LMU Director of Assessment, Laura Massa redacted each, to remove any identifying information (personal names, institution names) and anonymized each, assigning only a manuscript number. Massa’s team produced an evaluation packet for each reviewer, with included a mix of some pre-workshop proposals and some post-workshop proposals; each reviewer was presented with a unique mix of proposals. Each proposal had attached a printed scoring worksheet. Three of the proposals were removed from the evaluation process so that they could be used during the norming process.

We then invited a group of regional librarians to participate by acting as evaluators, over a two-day period. The in-person review experience was held in a large room in the LMU library. We prepared the room for breakfast, lunch, and snacks on each day, and offered an honorarium for each reviewer. At the outset of the review process, each participant signed a form saying that they would honor the intellectual property presented in the proposals and not attempt to conduct any research in the proposals they read.

During the first day of the evaluation process, Massa spent the majority of the day norming the rubric. This entailed talking through each of the components of the rubric and describing what each measurement meant. The participants then attempted to score a sample proposal and shared their scores with the group. Massa led a discussion about the scores, to resolve any variation among the scores, so that each reviewer had a clearer understanding about what each possible score meant. The group then scored another sample proposal and then again shared their scores and discussed any variation, so that the reviewers in the end were scoring in the same way.

The rest of the first day and the full second day was given to quiet time for the reviewers to evaluate the proposals in their packet. The reviewers were not told which was a pre-workshop or post-workshop proposal; each proposal was in two separate reviewers’ packets. Massa was available for consultation throughout the review process, to help guide any reviewer with a question about how to score something to consider their previous norming discussions. When the reviewers completed their evaluations, they gave the entire packet and scoring sheets back to Massa and were excused.

Massa’s office calculated the scores for each Scholar and for the group overall, using a paired samples statistic and effect sizes in SPSS. Massa returned to us an anonymized report and data, showing that in each case the post- score was greater than the pre-score. This is indeed what we were hoping to learn, that what the Scholars gained from the workshop was evident in the writing about their research projects, and they had increased their mastery at every component that was addressed in the curriculum.

Linked here is the version of the rubric and the scoring sheet in use from 2022-2024.

My reflection on the use of this tool for assessing the program
Once Massa left the position and we worked with a new Assessment officer at LMU, the weakness of a rubric as an evaluation method became apparent to me. The norming process, ensuring that each reviewer is understanding each component and its measurement in the same way, requires a commitment on the part of the person leading it. The new person we worked with did not challenge the group as strictly as Massa, to come to an agreement for when a component should be scored high or low. Without the agreement of the group of what was truly exemplary (a 4) or was rather developing (a 2), many reviewers assigned them to the accomplished category, a 3. The reviewers used this middle score of 3 throughout their evaluations, and the result was that the statistics did not show any gains from pre- to post-workshop proposals. This was quite a striking change from the first few years where there was a clear improvement in mastery from the post-workshop proposals. It points to the necessity of needing a consistent and assertive leader in the norming process, to make sure that the rubric is used as intended. This is, in my opinion, a weakness in using this method for an important assessment.

In the first few years of using the rubric, we invited regional librarians to participate in person as evaluators, without requiring that they had conducted original research themselves, only that they were familiar with the process and interested in participating. Some of the evaluators did have experience conducting research, however, which produced an uneven group for the evaluation. This is also a weakness of using a rubric as an evaluation method. Seeking a homogenous group would have produced a more consistent evaluation. Practically, however, coordinating such a group within a geographical area is limiting.

The cost of this assessment tool
For the first few years of this assessment, the person running the rubric process was employed at LMU and we included a percentage of her time (salary and fringe) on the grant. After that person left, we used a subset of funds from the grant to pay for the time of the person to run the process.

In addition to paying for food throughout the day, we paid for parking and an honorarium for each librarian participating in the two-day assessment.

Earlier posts in this series:
Introduction post, Confidence scale, Research networks of the Scholars, External review, Post-workshop survey

This post is part of a series, describing the assessments used to develop the Institute for Research Design in Librarianship (IRDL).

One month after the Summer Research Workshop was complete, we emailed them an anonymous link to a Qualtrics survey that took about ten minutes to complete, to ask about their workshop experience. LMU’s IRB reviewed the protocol ahead of sending it to the group.

The survey covered for areas of concern: the curriculum, instructors, their time at the workshop, and their cohort. The questions about the curriculum were to gather information about the amount of pre-workshop activities, how challenging they found the curriculum, how appropriate the curriculum was for their current needs, and the time set aside for hands-on activities and exercises.

We asked the Scholars to rate the instructors on clarity, helpfulness, and approachability.

We asked about the time commitment needed to participate in the workshop, and if they had sufficient support from their home institutions to focus exclusively on the workshop during the two-week period.

We were interested to learn if the workshop design impacted how each Scholar felt about belonging in a cohort. We asked if they felt part of the group, if the felt comfortable contributing to discussions, and how likely it was that they would stay in touch with at least one person in their cohort.

To end, we asked two open-ended questions for them to suggest any improvements for future workshops and if they had any other feedback about their workshop. Linked here is the version of the survey in use from 2022-2024.

My reflection on the use of this tool for assessing the program
I expected that this survey would give the Scholars the opportunity to tell us anonymously some critical things that they might not have told us in person (or via email) about their experience while the workshop was underway. I wanted to give the Scholars a bit of time after the workshop to mentally process their experience before responding, to allow them time to pause and reflect before we sent the survey.

We did seek their more immediate feedback during the workshop, even setting up an anonymous form that I checked every day, with this as the question: “Got something to say but would rather do it anonymously? We want to hear about it. We will check this once a day during the IRDL workshop. This form will collect a timestamp and the feedback you provide, that’s it.” From that form we learned that they felt as if they needed more one-on-one consultation time with experts, a suggestion for crowdsourced notes, rather than taking them individually, and that they loved the prompts for the morning sessions that were Scholars-only spaces to chat about non-research topics.

I expected from this post-workshop survey that we would learn that the workshop was too long, at two full weeks, but we heard quite the opposite (!). Over the years, the general feedback received was that the Scholars wanted more time for their expert consultations and wanted more down-time with their cohorts to process what they were learning. Time together, even with two weeks carved out specifically to think about research, proves to be too little for those seeking to begin their research endeavors.

I honestly did not expect the impassioned compliments we received in the open-text responses, one being, “Overall, this was a super impactful experience. I would HIGHLY recommend this program to anyone who wants to do their own research. It did exactly what I hoped it would do…I feel prepared to complete my research project!” I’m glad we provided a mechanism to receive this kind of feedback, too.

The cost of this assessment tool
Because LMU has an institutional subscription to Qualtrics, no grant funds needed to be used for the assessment. No specialized software was needed for the analysis of data.

Earlier posts in this series:
Introduction post, Confidence scale, Research networks of the Scholars, External review

This post is part of a series, describing the assessments used to develop the Institute for Research Design in Librarianship (IRDL).

When we initially developed the components of the program, we could only imagine what the experience of it would be like for the Scholars. We hired an external evaluator to be an observer throughout the time of the Summer Research Workshop, asking that person to focus on how the Scholars engaged with each other, the curriculum, and the environment we created during the Workshop. We requested immediate feedback during the workshop, if the evaluator noticed something that should be changed, and a written reflection after the workshop, to provide us with their thoughts about how we might improve the design of the future workshops.

In the first year of the program, we hired Linda Hofschire of the Colorado State Library to act in the capacity of external evaluator. In addition to observing during the Workshop, we asked her to review some of the data we gathered (confidence scale data and proposal evaluation data), to provide her perspective on how it may impact future iterations of the Workshop.

After the first year, we hired Nina Exner, an academic librarian pursuing a PhD, as the external reviewer, and asked her to continue in the role every year after, so that she could observe and comment on the changes we were making over time, in the curriculum, cohort-building activities, and environment. We continued with Nina even when we moved to the online environment (IRDL Online, 2021-2024). In addition to evaluating the program, Nina acted as a research consultant during the Workshop, providing one-on-one consultations with the Scholars on their research design.

My reflection on the use of this tool for assessing the program
When we designed the program, I thought carefully about the environment we were creating, wondering about things like the pacing of the Workshop, to make sure the Scholars had enough time to absorb the material and reflect on it, so they felt empowered by what they were learning and not overwhelmed. I thought about how the setup of the room might affect the interactions the Scholars would have, in formal hands-on group exercises as well as informal chatting during breaks. I thought about how their energy may be sustained throughout the day, and planned for a variety of snacks, yoga breaks, and walking breaks, so that they could retain their focus on the learning content when needed.

Having someone whose sole role was to observe the program in action, to look out for the Scholars during the workshop was reassuring to me, to make sure that they were being cared for as I expected. Based on the evaluators’ observations, we nimbly adjusted things for comfort and learning during the workshop and then later considered their written reflections for the design of the future years of the workshop. Over time we noticed that their written feedback suggested minimal changes, for a better impact; overall, the things they observed and reported on was what we hoped for the Scholar experience.

Working with the same evaluator year after year was practical for us, in both the continuity for the program, having someone to notice if a change we made based on their recommendation worked as they imagined it would, as well as interpersonally, for developing a long-term professional relationship with an expert in our field.

The cost of this assessment tool
Travel to and from Los Angeles, housing and food during the workshop, and an honorarium.

Earlier posts in this series:
Introduction post, Confidence scale, Research networks of the Scholars

This post is part of a series, describing the assessments used to develop the Institute for Research Design in Librarianship (IRDL).

Each cohort of IRDL had their own communication characteristics, from one in-person cohort meeting at night to review the day’s notes, using a whiteboard wheeled into a group study area, to another cohort chatting virtually, using a Slack channel. These informal collaborations reinforced learning and facilitated the development of a community of practice. We studied the size and strength of the research networks of each Scholar, using social network analysis, to determine if participating in IRDL influenced the evolution of their personal networks during the IRDL year.

Since we were interested in observing any changes over time, we administered an ego-centric network questionnaire four times during the year-long program. We used the free and open-source survey software customized for social network data collection, EgoWeb 2.0, for this assessment.

We installed the EgoWeb 2.0 software on a server (I used Linode) so that the questionnaire could be administered via the internet. LMU’s IRB reviewed the protocol ahead of Time 1. A week before the workshop began, I sent a personalized email with their individual link to the first questionnaire to each participant, to be completed before the first day of the workshop (Time 1); this questionnaire took about 10 minutes to complete. I released the second questionnaire (Time 2) via email on the last morning of the summer research workshop and gave the participants time to complete it that day. Six months after the workshop ended, I sent the third questionnaire (Time 3) and right after the program completed, at the one-year mark, I administered the last questionnaire (Time 4).

In our results (Kennedy, Kennedy, & Brancolini, 2017), we observed evidence that a small network at the outset expands throughout the year of participating in IRDL. In the figure, a visualization of a participant’s research network during their IRDL year, we see a dense cluster of people at Time 2, with the small triangle icons representing the people the participant met during the workshop and had frequent, reciprocal conversations about their research.

Change in a research network of an IRDL Scholar, over the course of one year

My reflection on the use of this tool for assessing the program
Since this was exploratory research, I wasn’t sure what to expect from the data, or how it would inform the design of the program. It was clear, however, from early on in our analysis that the program design was impacting the networks of the participants by being in a cohort, moving through a process at the same time.

In informal conversations throughout the Summer Research Workshop, some of the participants found like-minded researchers who were using similar data collection methods or working in the same functional area of the library. We encouraged these conversations by frequently offering research-focused ice-breaker questions that one participant could ask another, to help foster communication among the cohort. We also prompted the participants to consider who in the cohort or their workplaces at their home institutions they could talk to about their research. We offered a script/outline that the participants could use to talk to their library directors about their research once the summer workshop was completed.

The participants spontaneously noted throughout the year that being prompted to consider who was in their research networks was impactful for them. I also appreciated that the participants set up their own after-the-workshop communication mechanisms, to stay close to those in their cohort to continue seeking and offering help throughout the year. It has been fun to discover many cross-cohort collaborations, some of which have led to major publications in our field (notably Fullington, West, & Albarillo, 2020). An ideal future research project would be to evaluate the networks of the participants after a set period, after completion of their IRDL year, to note how many relationships persist and find out how they are thinking about those connections.

The cost of this assessment tool
To secure space on a Linode server and hold the small amount of data from the questionnaires cost $5.00 per month. Analysis was completed using R in RStudio Desktop, which are free to use.

Reflections on Practitioner Research: A Practical Guide for Information Professionals. Lee Ann Fullington, Brandon K. West, and Frans Albarillo, eds. Chicago, IL: Association of College and Research Libraries, 2020.

Kennedy, Marie R., David P. Kennedy, and Kristine R. Brancolini. 2017. “The Evolution of the Personal Networks of Novice Librarian Researchers.” portal 17(1): 71-89. https://doi.org/10.1353/pla.2017.0005

Related research, not an assessment of the program, per se. Luo, Lili, Marie R. Kennedy, Kristine R. Brancolini, and Michael Stephens. 2017. “Developing Online Communities for Librarian Researchers: A Case Study.” College & Research Libraries 78(4). https://doi.org/10.5860/crl.78.4.512

Earlier posts in this series:
Introduction post, Confidence scale