So, you're trying to run ARENA at your university group.
Case studies from 6 universities
This was previously a semi-public document, and I thought I’d just make it more visible so it’s easier to link to people.
In case you don’t know what the ARENA (Alignment Research Engineer Accelerator) program is, learn more here.
If you have previously attempted to run ARENA in any capacity, please contact me to add your perspectives to this post!
Contents
Summary
Stories
Alex Semendinger @ AISST
Ionel Chiosa @ MAIA
Gracie Green @ OAISI
Theo Farrell @ Durham
George Ingebretsen & Nelson Lojo @ BASIS
Jeremy Kintana & Will Anderson @ WAISI
Summary
Successful programs had two things in common: (1) intense time-commitment (sprints rather than marathons), and (2) at least one capable TA, typically a PhD student with relevant background.
In AISST’s case, a focused and dedicated participant who spent ~8hrs/week over the course of a semester could typically go through Chapter 0 and half of Chapter 1. Use this to set realistic expectations, and consider modifying the curriculum as needed.
Consider how you will handle participants who start at different knowledge levels, and who go through the curriculum at different speeds. Will you keep everyone synced up or accommodate for different levels of progress? See MAIA’s case study.
Alternatives to running your own in-house ARENA program: Directing dedicated individuals to the ARENA Slack, finding the channel named “study groups,” and letting them find someone else there.
AISST: Alex Semendinger
The AISST version has been pretty lightweight: show up on Saturdays from 10am - 6pm (or some subset of that time) and work through the material. Attendance is highest at the beginning of the semester; after midterms most people don’t want to spend their entire Saturday on this. And people would drop in and out, so while we had a schedule for which topics were covered each week, by the end of September it had little connection to what people were actually doing.
I think it’s been moderately successful overall. A lot of the value is just in having a dedicated place people can come work and having some TAs who can help them get unstuck if they need it.
I’m putting together a shortened path through the curriculum for the spring semester which touches on more topics at a lighter level. I’m hoping this brings more focus and keeps people working together longer. But historically I think spring semester has had less attendance than fall semester (there’s more “I have time to sign up for everything” energy and more new members in the fall), so I’ll have to see how this version goes.
Setting expectations: Average person who shows up at all is probably 1-2 sections. I think the rate is something like 1.5 days for each section, but that’s a very loose estimate and some sections are much longer than others. But I think a typical focused participant who shows up regularly will get through all of Chapter 0 and maybe half of Chapter 1 if you’re just going in order.
MIT AI Alignment is currently running ARENA as well — they’ve been meeting every day of January. I think that’s a better fit for the ARENA curriculum as-is, it helps for it to be the main thing you’re focused on for a continuous week or more. They’ve had some of the same issues with people dropping in and out and getting out of sync, but the most committed students have made a lot of progress and I think everyone who’s shown up has gotten good value out of it
Ionel Chiosa is the other TA and main organizer for the MAIA version. A few other logistical things they’ve done well: daily feedback forms to get a sense for how much progress everyone’s made + other comments, and (moreso in the beginning when students were on the same topics) short lectures at the start of the day to introduce the topics and flag any potentially confusing concepts before they come up.
MAIA: Ionel Chiosa
Format
We aimed to run ARENA over 4 weeks of January, in two sections of about 10 participants each. Section 1 was supposed to meet MTWTF for 5h from 10 AM to 3 PM, while section 2 met for only 3 days of the week, on MWF, also from 10 AM to 3 PM.
The goal for Section 1 was to cover all of Chapter 0, almost all of Chapter 1, and all of Chapter 2, with the hope that some people may be able to also cover the new evaluation chapter. For section 2, given that the group was meeting less frequently, we aimed to cover all of Chapter 0 expect for VAEs and GANs (last section), the first 34 sections of Chapter 1, and most of Chapter 2, but with less rigor.
We had 2 TAs: Me and Alex. My background is somewhat more mathematical, while Alex has more experience in software and industry, so our skillsets were complementing each other pretty well.
The format of the meetings looked something like:
Get everyone in one room for a 10-15-minute morning presentation that should give a sense of what the content of the day is all about.
Everyone starts working on their section. TAs check on every participant every 1h or so, and answer questions when needed.
We had a lunch break around 12:30, where we offered participants food for free. Some people preferred a more social environment, while others were simply picking up their food and going back to their laptops.
Everyone gets back to work until 3 PM. We remind participants to complete a feedback form before they leave.
We were also trying to match pair participants into teams of 2 every day. This was supposed to encourage a more collaborative environment and make ARENA overall more enjoyable.
Results
For the first 2 weeks, we were able to keep everyone on schedule. That is, most participants in section 1 completed half or more of Chapter 0, while participants in section 2 worked on the interpretability sections of Chapter 1.
However, after this point we had issues with attendance pacing. We were unable to keep everyone on track, so pairing participants was no longer an option, and the morning presentations had to be done more frequently and in smaller groups.
OAISI: Gracie Green
You can find more information on what we did at our website. I've also attached the schedule.
We housed our participants in Oxford for two weeks, provided breakfast and lunch on working days (often dinner as well during socials), and took people out for lots of socials.
I think the thing that went particularly well was our choice of participants - they were all fantastic, got on really well, and feedback often commented on the warm, friendly atmosphere we'd created. People also really appreciated going on a walk during the 3pm break, significantly more than I expected! They liked that we gave them 1:1 channels with one of the organisers so that they had a clear space to ask questions and discuss welfare concerns. They also really enjoyed doing the projects at the end.
I think people didn't particularly like the RL section, they much preferred the transformers. They also said they'd like it if we made time for academic talks at some point during the two weeks.
Durham: Theo Farrell
At Durham, we ran a shorter adapted ARENA course over 5 2h sessions.
There isn’t a bustling AIS community here (yet!), but we’re lucky to have one mech interp PhD (Patrick Leask) who taught this course.
Since this is quite a small course, I’m not sure how helpful my experience will be! Here’s what we found anyway.
The first session was an intro to PyTorch (session 0). This had around 20 people. It was very useful to have this to make sure everyone was on the same page.
Attendance dropped off over time as expected, but in the last session we still had 6 people, who were all quite keen.
We didn’t run it as a drop-in, but more like a tutorial where Patrick would walk through the notebooks and participants would follow along. We’d start with an explainer of the concept we were about to implement, and then participants would follow along in the notebook as Patrick went through it. At some points there’d be an exercise that participants would have time to complete on their own, while Patrick went and helped anyone who wanted it. We’d have a break in the middle too.
We recorded the sessions, which was useful because sometimes people couldn’t make one week and it was useful to have recordings as a resource to catch up.
BASIS: George Ingebretsen and Nelson Lojo
George:
I think some issues were that:
We did it during school so people were too busy.
We didn't have food or snacks.
We didn't make it clear that it’s okay to fall behind schedule - people would fail to get the weekly assignment done and then drop off because they thought they had to be at a particular point to attend.
I believe it only lasted a few weeks before attendance dropped to ~0 unfortunately.
Nelson:
Another point I think was a pitfall was the lack of advertising throughout the semester. Generally, I think we gave the event too few resources (e.g. organizer time, incentives, marketing) to be able to build a cohort.
As for good points: the people that were able to discuss interests outside of ARENA (even if they were still research related) had very high retention, even to the point of maintaining contact after the "program" ended.
WAISI: Jeremy Kintana & Will Anderson
Jeremy:
Whenever we ran ARENA, we didn’t consider it to have much logistical overhead, and sorta assumed that it would run itself. This was a big mistake! ARENA is hard and running it takes dedication. If you do decide to run this, consider it to require as much (or more) organizer time and attention as your normal programs (e.g. reading groups, intro groups, events).
Over the summer/winter breaks, we attempted to have people pair up and coordinate, with a single facilitator doing check-ins on the pairs. This fizzled out. People have summer/winter break plans, and while something like ARENA can be self-directed, it’s way more challenging that way.
We attempted to run co-working sessions during the semester, where we’d have two different groups of about 3 to 4 people each, meeting for 2 hours per week. This also fizzled out as people got busy over the semester, and progress was extremely slow — I’d be surprised if people completed a single section. In hindsight, we should’ve consolidated the groups, and spent a lot more time than just 2 hours.
Will:
I believe a big part of the trouble we had running ARENA came from making it optional and having little to no way of enforcing that people make progress. For every ARENA session, people had to make the conscious decision to attend (something which takes more effort than we realize). ARENA is hard, and can be demotivating, so when people have to decide whether they will work on it each week, it’s easy for them to skip.
In our intro and reading groups, we make continued involvement in the program conditional on attendance. We use a yellow-card red-card system. If a participant needs to miss a session, they must send a message ahead of time, and will then receive a yellow card (unless it is for some major unavoidable reason). If they do not message ahead of time, they receive a red card. Receiving two yellow cards or a red card triggers a meeting with the facilitator. We’ve found that this helps a lot with attendance. Implementing some system where people have to consciously opt out rather than consciously opt in can go a long way.
Cover photo by Annie Spratt on Unsplash.