Author: Kurt Luther

Since our Crowd Lab builds and studies crowdsourcing systems, I often think about how our research might apply to ourselves. Research labs can have surprising overlap with online labor markets like Amazon Mechanical Turk and Upwork, in both positive and less flattering ways. Shared characteristics might include transient workers, mixed levels of experience, diverse motivations, technology-mediated interactions, and even geographic distribution (my own group is split across two campuses).

I became more aware of these similarities as my own lab grew. After few grants came through around the same time, I was suddenly fortunate to have money to recruit more grad students, and quickly. Yet, I was also turning away good candidates for undergraduate research because I felt I was at my capacity advising five or even ten students per semester. These are good problems to have, but I wanted to make the most of my opportunities. I wondered: How might crowdsourcing research help me scale up my “crowd”?

Fortunately, some exciting new research suggested answers. Other crowdsourcing researchers, namely Haoqi Zhang and Michael Bernstein, have formalized their lab-building efforts into successful research enterprises, like Agile Research Studios (ARS) and the Stanford Crowd Research Collective, whose findings benefit the broader academic community.

When Haoqi and colleagues published their first paper on ARS at CSCW 2017, I was excited to adapt some of their techniques for my own group. I had already begun integrating agile methods into our project meetings and hired a part-time scrum master. ARS proposed a more comprehensive computational ecosystem, with a strong emphasis on both research and education. I was intrigued, and over the summer and fall of 2017, our lab became more ARS-like.

During this time, Haoqi and I had a few email and in-person conversations about how I was fitting ARS into the unique context of my lab and university. He invited me to visit Northwestern as part of his Agile Research University program. It was an opportunity to see firsthand how his students and colleagues put ARS into practice. I gratefully accepted.

Over a few days in November, Haoqi and his Delta Lab co-directors, Liz Gerber, Matt Easterday, and Nell O’Rourke, immersed me in the ARS experience. I attended lab meetings and project meetings, sat in on the interdisciplinary Design, Technology, and Research (DTR) undergraduate course, participated in pair research sessions, and asked lots and lots of questions. It was an intense, but illuminating experience.

I arrived at Northwestern feeling like I had a pretty solid grasp of the ARS process. I’d read the paper multiple times, implemented some of its components, and discussed it with Haoqi and his PhD student, Leesha Maliakal. However, I found that being there in person allowed for a much deeper understanding.

While Haoqi’s paper clearly outlined the mechanics of ARS, it’s harder to describe the culture that Delta Lab members and the DTR course cultivated. I can at least share some observations that impressed me. I watched students enthusiastically partner up with others whom they didn’t know well, but were eager to help. I saw students vigorously, yet politely, debating the pros and cons of a design alternative. Students bought lunch for their classmates and reflected together on their progress, with remarkable courage and vulnerability.

I discovered that Haoqi and his colleagues had made many small, but important decisions to foster this culture at their institution. Lab meetings begin with playful, positive exchanges. Breathing exercises during class breaks build trust and encourage reflection among students. Haoqi lets students borrow his office for meetings, and its spare, configurable furniture served a wide variety of needs. File cabinets became impromptu seats, and students gathered around an external monitor for a quick demo or critique.

I also noticed how the design decisions enabling ARS, both curricular and environmental, mutually reinforced one another. Students in ARS perform well because they’ve been selected from a large pool and passed a rigorous interview process, but this screening process is scalable because it’s delegated to students. Students tackle ambitious projects requiring diverse expertise because DTR is an interdisciplinary course that draws from multiple majors. Students benefit from peer mentoring and organizational memory because they get course credit for enrolling multiple semesters.

During breaks and over meals, Haoqi and I talked about ways I could bring some of what I’d seen and learned back to my lab at Virginia Tech. We agreed that adaptation was essential. Each institution and group of researchers is unique, and brings its own affordances and constraints. We also agreed that a gradual ramping-up was the best way to proceed, so we talked about prioritizing different pieces of the ARS model.

I left inspired and with detailed plans for the next semester, which I’ve recently implemented. These included a more flexible use of physical space, a shift in meeting structures to encourage undergraduate mentoring, and new digital resources for helping students with long-term planning. We’ll approach this semester’s changes like we have in the past–as an experiment, subject to iterative design and formative evaluation. Based on our past success with ARS techniques, and my experience visiting Northwestern, I’m optimistic.

With their focus on culture as well as process, attention to detail, and systems thinking, Haoqi and his team have created something special. I look forward to capturing some of that magic, to build new connections between my crowdsourcing research and my own work practice.