Peer Assessment in Cyber Fellows: A conversation with Professor Aspen Olmsted
"Everyone thinks online education doesn't work. But the truth is that we can actually deliver better education online."
Professor Aspen Olmsted is the director of Tandon Online's Cyber Fellows program and an adjunct in Tandon's Computer Science and Engineering department. He obtained his Ph.D. in Computer Science and Engineering from The University of South Carolina. Before his academic career, he was CEO of Alliance Software Corporation, where he developed N-Tier enterprise applications for the performing arts and humanities market. Sr. Instructional Designer, Jay Leibowitz, recently met via Zoom with Mr. Olmsted to discuss his work implementing peer assessment in Tandon Online's Cyber Fellows program.
Aspen Olmsted: I’m the Program Director for Cyber Fellows, which is essentially an online delivery mechanism for our Masters of Science in Cybersecurity. I'm not a cybersecurity person in the traditional aspect; I'm a software engineer. I spent about 25 years in the software industry before I came to academia, building software for performing arts centers and museums. When we build software, we always need to take into account those core competencies of cybersecurity. According to the Bureau of Labor Statistics, in four years the number of open positions in cybersecurity will be more than double it is today. Our goal with the Cyber Fellows program is to train the future cybersecurity workforce.
JL: We've got a lot of people from various industries in the program, which I think figures into the benefits of using peer assessment in these particular course offerings. Can you tell us a little bit about the typical Cyber Fellows cohort?
AO: Cyber Fellows students tend to be working adult learners looking to transition to a more cybersecurity-focused role in their organization or a higher, more technical position within the cybersecurity industry. About a quarter of the cohort has an undergraduate degree in computer science, which means that 75% of them don't. Another quarter is in some sort of technical industry, like management of information systems or a traditional business with some technical focus. The remaining half comes from other disciplines, which isn’t uncommon in computer science. Most people working in computer science don’t have a formal education in computer science.
JL: I know a lot of cybersecurity folks who I worked with in my previous position in the legal space had years of information technology and support experience and found their way into cybersecurity, earning certifications along the way. That seems to represent a good portion of the Cyber Fellows cohort.
AO: Right. We’ve certainly got folks coming from the governance or the policy side of their organization, which is more related to law. Incidentally, we also have a masters program for professionals with NYU Law called Cybersecurity Risk and Strategy that is really designed for that non-technical side of cybersecurity. The Cyber Fellows program at Tandon is a very technical cybersecurity degree, which can be a challenge for people who don’t necessarily have the technical background. That said, we’ve all experienced cybersecurity in some form, either as a victim of a cyber crime or through the emails we receive from our IT departments asking us to take steps to ensure that we don't leak information. Everyone has something to contribute and part of the idea of peer assessment in the Cyber Fellows program is to draw that out so that they can bring it to the table.
JL: Can you share an example of a unique perspective on cybersecurity from an industry that we wouldn’t normally think of as relevant to the subject?
AO: I had a student who worked in the Water department at my old university. She was a cybersecurity major in a master's program at work on a research project that I’d assigned where the students assess risk in their industry. This particular student had a hard time figuring out how to apply what she learned to the Water department. She worked in a domain where we're all very vulnerable. We all rely on clean water. Now, I have friends in the Netherlands who work on the cybersecurity team for the Water department there and, if you can gain access to their system, you can flood the entire city of Amsterdam.
JL: Yikes.
AO: You can seriously affect people's lives. You can kill people. Students don't always realize the knowledge they have access to. Part of the peer review process is to draw that out and allow them to share their unique perspectives with each other and gain confidence in their understanding of how they can uniquely contribute new knowledge to the world.
JL: I think that's an extremely effective example of how peer assessment can benefit this particular cohort. When I started to work with you on designing your Introduction to Operating Systems course for Cyber Fellows, one of the things that I really appreciated was your grasp of online learning. I think the school in general benefits greatly from your work with Tandon Online, because you've got a really firm grasp of what works and what doesn't. You take a practical approach, which I certainly appreciate. I wondered if you could talk just a little bit about your experience teaching online before Tandon, and maybe how it's changed with your work here.
AO: We live in a society right now where you turn on the news and every day you hear all about the K- 12 problem of school for Fall 2020. Should students be online? Should they not be online? Everyone thinks online education doesn't work. But the truth is that we can actually deliver better education online.
JL: I most certainly agree. It just takes a little bit of preparation.
AO: I'm not criticizing our face-to-face cybersecurity program. It's terrific. But we're doing all these things in online education that would have been hard to do in a traditional face-to-face or brick-and-mortar classroom. I think about this every day. Online students inherently understand the idea of iteration. You take your first pass, you try to understand it, and maybe you say to yourself, “let me reread that again and see where I don't understand.”
JL: Asynchronous learning in a nutshell.
AO: Exactly. It’s a privilege in our society today to have that experience. In the future, we're not going to have someone standing over our shoulder pointing out what we don’t understand or what we’re doing wrong. I think online education can be way better than brick-and-mortar. I'm not saying people shouldn't do brick-and-mortar, but a mix. It’s good for all humans.
JL: How did you come to this conclusion in your career as a teacher?
AO: I started teaching online probably 11 years ago in my old university. The College of Charleston is on this little finger in South Carolina we call “The Peninsula” and it's a pain to get to at rush hour, because everyone's vacating the city to go out to the suburbs. It was difficult to get students to come to our classrooms at 7pm, when we ran our face-to-face classes.
JL: So, you went online?
AO: Yes and I had no idea what I was doing! You know, the first time you think it's easy. You’re just doing what you normally do online. I started recording my lectures using Camtasia and experimenting with it.
JL: I’ve seen your Camtasia lectures from that time. They’re not that different from NYU's standard solution, Kaltura Capture: mostly desktop screen captures with picture-and-picture, is that correct?
AO: Well, my first pass was just audio over PowerPoint. Then I went to the “bobblehead-in-the-corner” and I learned that the bobblehead communicated a more personal experience for the students.
JL: I agree with you there, too. It's an ongoing debate, but I think most people agree that a small video feed of your professor in the corner of your screen goes a long way towards engagement. Students feel like they’re having an authentic experience.
AO: That’s because it is an authentic experience. After I started doing the bobblehead, I tried to figure out ways to do online labs, which is very hard. But, after you've taken an online class, you realize that it’s okay to read the instructions, do your best to understand them, and make an attempt. Then what happens? You get feedback and you go back and read those instructions and you try it all over again.
JL: When I first started working with you on your Intro to OS course, I noticed right away that your term project was structured in a way that seemed to lend itself to a peer assessment process. I wondered if you'd used peer assessment in online learning previously?
AO: Well, sort of. I didn't know I was doing it. Students would present to each other on video or in class, but not as a formal way of reviewing each other. There are several reasons why students perform better for each other than they do for me. One is that they're more concerned about what their peers think of them. That raises the bar in a way that's terrific. I learned that providing multiple steps just brings everyone up, too.
JL: That’s what we mean by “scaffolding”.
AO: Exactly. And providing them an opportunity to see each other and interact early on in the course helps a lot. In research, you tend to have a secondary author, someone to bounce your ideas off of. I just didn't have the time for every student and I wanted to create opportunities for students to have someone else to run their ideas by. I used to make it a requirement at NYU that everyone had to be a secondary author on everyone else's paper and a primary author on their own. I still like that idea.
JL: I was delighted when I looked at your Operating Systems course and saw that you had already designed a scaffolded term project. I immediately saw that this was a great opportunity to incorporate peer review and that the Cyber Fellows students would benefit from it. I know that it continues to be something that you encourage in course assessment designs for the program. Can you talk a little bit about the operating systems term project and the way it works outside of the peer assessment component?
AO: Sure. Operating systems is a funny course where we learn about all the data structures that an operating system uses to essentially abstract away the hardware. Traditionally, we talk about Windows, OSX, or Linux. But something like Salesforce functions like an operating system, too. The goal for the students is to hypothesize a problem within their domains of knowledge and posit ways in which they can contribute new knowledge about future operating systems to solve that problem. Some of them will be excited about becoming a problem solver, instead of just learning the competencies. We want students in Cyber Fellows to take their knowledge about the world, apply it to new problems, and find new solutions. The goal of the project is to produce a work that documents some evidence to solve a problem. Students aren’t graded on whether or not they’re successful. It's totally okay that they hypothesize a bad solution, because they still learn. That's core to the iterative aspect of learning this way. You try something and you learn from it. I worked for 25 years of my life in the performing arts and I know about problems in that industry that no one else understands at the same level.
JL: We have that in common.
AO: We do! Other students know about other things. Maybe they’re in banking, finance, manufacturing, or water. These are all domains that I know nothing about. But I do understand cybersecurity and how to apply the scientific method to problem solving. It’s really about getting our students to go through that process.
JL: I know that in the Cyber Fellows program there are people working in many different industries, from financial to zoology. That's one of the things that I thought was exciting about your project. It was a clear opportunity for students to really bring their own experience and their own knowledge to the table. You had already scaffolded the project, so that it steps you through writing each part of the paper throughout the semester. It’s a great design that lends itself to peer review. But, before we incorporated the peer review system, students were just submitting that work to you, right?
AO: Yes. Before that, I would have to explain the steps and where they should be, but I didn't have a formal process of doing that. I would make them present more often, rather than just at the end for the final. That was a smaller group that we're talking about, 20% the size of Cyber Fellows, where our classes are typically 150 students or more. It'd be hard to do that without some system to manage those assignments.
JL: And it needs to be an asynchronous online program. Not everyone can login at once and present to each other.
AO: That’s correct.
JL: Just to be clear, when we talk about scaffolding, each assignment is a small piece of the term project, and each assignment throughout the semester is a small step towards completing the final project. So, by the time we get to the end of the course, students just need to synthesize everything that they've done: wrap it up, fine tune it, and present it. Throughout the process, there are built in opportunities for students to receive feedback from their peers, which is very valuable, as peers are often working in different domains. Students also benefit from the self assessment phase of each assignment, as well, where after their peers respond, they can look back at their own work and revise it.
AO: Right. The first submission is to choose a specific problem in a domain that they are familiar with and then hypothesize a solution to that problem. Not a problem, a domain. “I'm going to work in the water industry.” That's all they need to do. And then they go to Google Scholar and find three academic papers that solve problems in that domain. The purpose here is twofold: (1) It slowly introduces them to narrowing down to a problem which (2) gets them to think about Literature Review. I say to them, “You can change this at any point. If you don't like this domain, that's fine. If you don't like the papers you chose, that's fine, too. This is just your idea. They submit a paragraph that describes the domain along with three references to papers. Finally, they review each other and reflect on their own work. Next, they do some preliminary evidence gathering, some testing, and then they produce just a small extended abstract paper. I tell them, “This is just your flag in the sand. You're just establishing yourself around this piece of work.”
JL: And those are the first and second assignments. What’s next?
AO: The following week, they narrow it down to a specific problem and a hypothesis to solve that problem. After that, I give them two weeks to think about how they're going to measure their solution. There are two big problems we're overcoming here. One is: I can't solve any problems! And the other is: Do I really need data? These are the two objections students tend to have. I try to explain to them that they know things that I don't know about domains! “Use that,” I tell them. “Don't throw it away. Don't throw away your life experience. We're here to give you competency in cybersecurity that you can add to your life experience to go out and solve problems for the world. We are scientists. We want data. We want proofs. But, it doesn't have to be perfect.”
JL: What’s the next assignment look like?
AO: I have them submit a chart. It could be a table or a figure that compares this made up data to what they expect to see happen. And that sort of concludes our first phase. They get a couple weeks, but they don't submit the final paper right away. Instead, they start building up the pieces of the paper. They do the introduction section, the related research section where they do the literature review, and then their empirical evidence section. In the final weeks of the course, they put it all together into a 15 minute PowerPoint with voiceover slides. (PowerPoint’s free for all students. It works on Mac and Windows out of the box and you can record your voice in the application via MP4.) I have them upload it to NYU stream so they can caption it and share the link in Peergrade, along with a two-page paper.
JL: Thanks for explaining the whole process. It’s such a great design. Each of those peer review steps are moments for feedback where you can really get into those phases for review. Let's get down to the nuts and bolts of what that looks like for a student. I already know the answer, but I'm going to make you say it, if that’s alright?
AO: Sure!
JL: I'm if I'm a student and I'm reviewing somebody else's work, what is expected of me? What does that experience look like? I know a lot of students can get anxious about peer assessment, particularly because the platform we’re currently using for this process is called Peergrade. They sometimes think that means that there’s a chance that their peers are going to tear them down and they’re not going to succeed. But that's not really what it's all about at all, is it? Let's talk about that anxiety.
AO: This I had to learn over a couple iterations of running peer review. I have a webinar always before the submission is due. Something I say every week is, “take what you can from the reviews.” We all communicate differently. I have a 13 year old son and everything out of his mouth is harsh! His mother is always offended! But, that's just how he communicates. He’s not angry. Some of us just communicate that way.
JL: Particularly online.
AO: Yes, absolutely.
JL: But, the idea is that - in asynchronous online learning - you don't all get to be in the same room together and discuss each other's work. So this is meant - quite specifically - in many ways to address that. I would argue, too, that this type of thoughtful peer assessment plan can really enhance the experience of synchronous classroom discussion. Because, when you all ultimately login to Zoom or get in the classroom, the conversation has already begun.
AO: The students are learning about how to do scientific research to solve problems. They're also learning about the process by reviewing each other and learning about other people's ideas. I'll often look at the students' work and discover brilliant ways of thinking about things that I wouldn’t have arrived at.
JL: A great benefit to you, as a teacher.
AO: And they're also learning about each other as people. And finding people with similar interests is part of what happens in a face-to-face class that can also happen online this way.
JL: And that's one of the primary objectives of the Cyber Fellows program: to network and learn about each other's work across industries. I was wondering if you could just talk a little bit about your thoughts around rubrics and how to design them, particularly in terms of peer assessment and the work that you're doing.
AO: Sure. We have rubrics for each assignment. I've turned it more into participation, because this grade is only 20% of the final grade. And when you think about something like that. I think that's a good size for peer review assessment
JL: That’s important because it goes a long way to address the issue of anxiety around being graded by your peers.
AO: Maybe 30% is fine. But we shouldn’t make this 50% of the grade. We learn from doing these steps. There will be students who do half the reviews and half the submissions and students who do every single one. And we want more of them to do every single one. It isn’t about doing them. It’s about learning from each other through that process. It's all about participation. Each student has 6 opportunities to submit with 18 opportunities to review. After the last assignment before the final, I do a grade on the peer assignments based on participation and then I leave the last half of it - the last 10 points - for the final submission. You want everyone to participate and learn from each other to be successful.
JL: Do you think that using rubrics is important? Do you think it's necessary for peer review?
AO: Yes. It goes a long way towards helping the students know what they're reviewing each other about. Some people don’t always put a lot of effort into the written reviews and so the rubrics force them to think analytically and put more effort into their structured response.
JL: The rubrics are interactive in Peergrade, too. It isn’t just a table that's introduced at the beginning of the assignment, like you might normally include within NYU Classes. In Peergrade, students can click through the criterions within the rubric and then provide written feedback before they submit. It encourages specific feedback, as opposed to sort of just leaving a door open, where people can walk in and tear each other down. This is a very specific framework for people to feedback on each other’s work.
AO: That’s right.
JL: What has been the overall response?
AO: I've not had any negative feedback about peer assessment. I think that's a strong, strong message. We've tried to work with instructors in other courses to get them to use peer review without enough coaching in advance.They sort of just threw the assignments up there and the students reviewed each other without thoughtful rubrics or scaffolding that builds to a bigger assignment.
JL: It doesn’t sound like that would be particularly effective.
AO: It isn’t. Students like doing the research. I gave an option for a project in Cyber Fellows where the idea is that students visit a research project every semester throughout the program. Students come up with a hypothesis to a problem in the first course of the program and produce a small work that's peer reviewed in that course. Then they go on to the next course for the next step. The students want to do this. They can update their work halfway through the semester with a couple paragraphs on how they think the current course will reflect on their overall idea and then peer review each other. A couple weeks after the semester, they update their paper again and peer review each other again.
JL: You've really run with this concept in so many wonderful ways since we got started with it. Since you've got so much experience now running these courses and iterating your designs around scaffolded peer review, I think our community would benefit from any design strategies or best practices that you can recommend. Your term project in Introduction to Operating Systems was a great candidate for scaffolded peer assessment. It's really a best practice: to find a big project and break it down into small parts. What other strategies or recommendations would you have for other people considering implementing a similar assessment plan?
AO: Most of the cyber fellows are adult learners who bring lots of experience, but they also bring lots of anxiety. I think breaking down a big project into small parts keeps them in school. Lots of people come back to school, but it's really easy to think that you’ve got a bad idea and ditch it. Peer review really helps with that. If you introduce this big research project and you tell everyone to hand it in at the end, that’s scary. But if you say, “pick a domain that you want to solve a problem in and go find a couple papers about it,” that's not scary at all. I haven't done any research on this, but I believe this helps with retention and I see there's so much less stress about the paper. My recommendation is that, anytime there's a big project, whether it be a research project or report, break it down into as many scaffolded steps as you can. The more the better. Create a Powerpoint presentation outlining the submissions and review it each week. Repetition and iteration are our friends, especially in classes like ours which have 160 people and growing. It helps the students to hear it again and again and again and again because they don't have confidence in this area, and that's what the scaffolding steps are for: to build up confidence.
JL: Great advice.
AO: The other recommendation I have is to encourage students to take a deep breath before they read those reviews. Professors get reviews every semester and we always have to take a breath before we click on that link with the comments. And you go, “Okay, it's gonna be okay. It's gonna be okay.”
JL: It sort of goes against what most people have learned about online in general, which is to never read the comments. Here, we're encouraging everyone to comment and read the comments.
AO: Just take what you can from every comment. It's something I've learned in online education and adult learning and you said it earlier: We tend to be nasty to each other online. I try not to take anything personally. I try to help and respond with as much information and as little tone as I can. I try to encourage the students to do that in their reviews and not take any of it personally. They're getting free advice from their fellow students and they're building a work that's their intellectual property so they should be happy to get free help.
JL: They're also building a network of peers. The IDM Media Law course that I designed for our IDM MicroMasters program on edX features an abundance of overlapping peer assessed assignments. There, we have reminders to be respectful and encouraging to each other for each of those peer assessed components. I think critical feedback in this day and age is a really important skill to develop. Students need to learn how to provide it without being negative.
AO: That's very true. Young students are used to being public and worrying about that more than we were, I think. My daughter's always concerned about her peers' assessment of everything she does online, where our generations didn’t have to deal with that. And so getting a pass, I think, is important.
JL: That's a good point. Cyber Fellows learners are working adults. If I'm teaching an online or blended undergraduate course, I’m dealing with a very different cohort, who is probably much more used to feeding back reflexively on each other's work. But I think we are talking about something different, which is a structured system with rubrics and phases, that's much more focused on the work itself. Anything else you want to add before we sign off?
AO: Cyber Fellows plans to grow to where our class sizes could be in 300 or 400 students. Peer assessment is a big part of that plan. Once you have it set up, it’s very easy to scale up, which is a huge benefit.
JL: Terrific. Thanks again for taking the time to speak with me today, Aspen, I really enjoyed the conversation and look forward to seeing how this aspect of the Cyber Fellows program develops.
Comments
Post a Comment