By:Jonathan Aldrich, Sumon Biswas, Steve Blackburn, Benjamin Chung, Youyou Cong, Alex Potanin, Hridesh Rajan, and Talia Ringer

The authors jointly served as hybridization co-chairs of SPLASH 2021, except Rajan who was the general chair of the SPLASH 2021 conference.

Introduction

Due to the COVID-19 pandemic, many conferences were held online starting in late March 2020, and several of us helped with the fully virtual offering of SPLASH 2020.  By mid-2021, however, vaccination was widespread and effective at preventing COVID infection and severe disease, and it became feasible to consider in-person attendance at conferences.  Polls of authors and the larger community indicated strong interest in holding an in-person version of SPLASH.  Still, as of October 2021 the US had not relaxed its travel restrictions on people from large parts of the world, and even beyond that some people would not be able to travel to SPLASH.  We therefore decided to offer SPLASH as a Hybrid conference in October 2021, offering both in-person and virtual attendance options.  We were also excited about piloting a hybrid conference, because there are many longer-term benefits if we can run them well, including supporting both in-person networking and accessibility for attendees who can only attend remotely.  We knew doing so would be hard, but we also hoped to learn valuable lessons.  SPLASH 2021 turned out to be an interesting case study as the first major post-COVID PL conference to go hybrid, and one of the earliest across the ACM as well.

Logistics

SPLASH includes several colocated workshops and conferences; these include the core conferences OOPSLA and Onward! (held along with REBASE and SIGPLAN Papers on Wednesday through Friday) as well as a number of smaller events Sunday through Tuesday.  The venue included 6 rooms, and we had 6 tracks running simultaneously a large percentage of the time.

We asked all authors to prepare video presentations of their work which we could use for mirroring; authors had the option to present in person or to have the video played in the live session at SPLASH.  Each session was mirrored 8 hours later, except that the last session of the day was mirrored 8 hours earlier (see illustration).  This solution ensured that the mirror sessions would be during waking hours in Chicago so in-person authors could attend the Q&A session for the mirror–-but also either the live or mirror session would be accessible during waking hours anywhere in the world.  Questions were accepted via Discord chat and were asked by the moderator, and people attending in person could ask questions live.  Questions were answered live by the authors whether attending in person or remotely via Zoom.  All questions were asked and answered at the end of the session, after 4 paper presentations, to reduce the cost of switching between presentation and Q&A in a hybrid setting (where it is more difficult than at an in-person conference).

On the main conference days we had 3 tracks devoted to papers from recent SIGPLAN conferences, including OOPSLA 2020, Onward! 2020, SLE 2020, DLS 2020, ICFP 2020, ICFP 2021, PLDI 2020, and PLDI 2021.  We invited all attendees of these conferences to present if they wanted to, but the condition was that they had to attend in person—all talks had already been given virtually, so this was a chance to do what they had missed previously.  A minority of authors overall took us up on this, but between all the conferences it ended up being a large number of papers and the participants were very enthusiastic about the opportunity.

All aspects of the organization were communicated to authors, attendees, organizers, and volunteers through web pages, emails, Discord posts, YouTube videos, and online training sessions. These were linked from the conference web page. Using multiple modalities and providing multiple opportunities to get the message was important to getting the complicated logistics to work out.

Benjamin Chung set up a Hybrid Table—with a large video screen, speakers, and directional microphone pointed down towards people standing at the table—to support informal interactions between remote and in-person attendees during breaks between conference sessions (see picture on the right).  The table was positioned in the hallway outside the main conference rooms, near to where food and drink were being served.

We set up a hybrid social program with either an Ask Me Anything or a PLTea session at every break.  Some of the Ask Me Anything interviewees were remote and some were in person.  Both were well-attended and seemed to contribute well to providing some social outlets that remote participants could join.

Since SPLASH was held in the middle of the Delta variant outbreak of COVID, we required all attendees to be vaccinated, and to wear masks except when eating and drinking.  No exceptions were made, which seemed prudent for one of the first conferences to restart after widespread vaccination.  Many people ate outdoors—the weather was a bit chilly but fortunately mostly clear during the conference, so a good coat was enough to make this feasible.  ACM required us to use their vendor, which imposed some delay (several hours) between registration and attendance—so it was not possible to register and immediately start attending.  Given empirical data showing the high effectiveness of vaccination on preventing the spread of Delta and mitigating severe disease, we were confident that this would be sufficient to avoid serious outbreaks, but some attendees were concerned about the possibility of a breakthrough infection and took additional precautions such as not eating or drinking in shared indoor areas.  In the end, the conference’s precautions were successful; there were no known transmissions of COVID at the conference, despite substantial indoor eating and drinking.

Overall, we had 237 in-person attendees and 573 virtual attendees.

Observations and Lessons Learned

Hybrid was a net win for SPLASH. Being virtual or hybrid was necessary due to COVID-19; otherwise many paper authors could not attend and present their work.  But those who could attend in person were very happy to be there, and seemed to get much more out of the conference than they would have if it had been a purely virtual event.  We are not sure SPLASH was better for virtual attendees than a purely virtual conference would have been—there were some benefits like being able to interact with in-person attendees via the Hybrid Hallway Table, but also drawbacks related to the more complicated logistics (we experienced more A/V glitches in SPLASH 2021 than during the purely virtual SPLASH 2020, which was probably inevitable due to the complexities of hybrid).  However, a major win for in-person attendees and not making things significantly worse (and maybe better in some ways) for virtual attendees is a win overall.

Center the in-person conference around in-person interactions. One of our key observations was that in-person attendees flocked to in-person presentations.  We expected the newest papers from OOPSLA and Onward 2021 to attract more attendees than the re-presentations of older work, but the opposite was true, because all of the re-presentations were in-person whereas some of the brand-new work was presented virtually.  Even a very well-done virtual presentation is at a significant disadvantage vs. a well-done in-person talk.  This is perhaps unfortunate—it means virtual presenters will never be on a level playing field with those who attend in person—but it does give us guidance about how to organize conferences.  First, encourage and enable people to attend in person and present in person where possible.  Some presenters at SPLASH 2021 attended in person but chose to have their video played; while some may have good reasons to do this, we should encourage all those who can reasonably present in person to do so.  A key part of this is enabling more people to attend—this includes providing financial support where needed (SIGPLAN does a lot of this already) but also things like providing childcare options, making sure the conference is accessible to people with different kinds of disabilities, and avoiding hosting conferences in countries that have travel bans impacting large parts of the world or large groups of researchers.  Second, group papers into separate in-person and video sessions.  This makes logistics easier—switching between video and in-person is hard—and it recognizes the reality that many audience members will want to attend the in-person sessions and avoids creating the extra churn of people moving between rooms to accomplish that.

Small hybrid workshops with a critical mass both of in-person and remote attendees can work great. An example of a great Hybrid experience at SPLASH was HATRA, a small workshop that started out maybe half and half virtual and in-person, but ended up attracting additional in-person attendees who sat in because it was interesting and had a core in-person group.  It worked well for a number of reasons.  First, there was a critical mass of people both virtual and in person, allowing both groups to feel they were a first-class part of the workshop.  Second, the small setting allowed us to use the same dedicated Zoom room throughout, making the physical room and the Zoom room feel like relatively seamless extensions of each other with people able to come in and out as during the usual workshop.  Third, the content was a mix of presentations both by remote and local attendees and discussions, which increased interactivity.  The discussions were done in breakout groups—some groups entirely online and some entirely in-person—followed by reporting back ideas from the groups to the whole workshop. This is a pattern that could be repeated in future events.

Another event that worked very well in hybrid mode was the Future of Conferences session, a panel with both remote and in-person panelists and audience.  It worked well again because of critical mass and the interactive nature (among panelists, but also later from questioners).  The one awkward thing was that the moderator was remote and it was hard for them to see when in-person panelists/participants wanted to contribute.  An in-person moderator would probably have been able to juggle better, but that wasn’t an option in the particular situation.  Another best practice for discussions is to dedicate someone to monitor chat who is separate from the in-person moderator, as it can be difficult to monitor both chat and in-person comments fairly.  Overall, the session went very well despite a few challenges.

There were other workshops at SPLASH that were successful as well, but they seemed to be more dominated by either in-person or online attendees and so were more like a conventional in-person or virtual workshop than a full hybrid event.  Future conferences should try more creative ideas for hybrid technical events; one example might be group hackathons that could work really well with both remote and in-person participants.

The Hybrid Table was fun, though it reached a limited number of people and we observed some challenges.  We had some great hallway conversations between in-person and online attendees via the Hybrid Table.  However, those conversations involved a limited set of people.  We should look at ways to encourage more people to participate, both in person and online.  Further improving the A/V experience (so everyone can hear without shouting) while still keeping the table in view of the conference social area could help. Worth noting that for virtual attendees, a directional microphone mounted on top of the TV stand was amazingly effective at making it possible to hear the speakers addressing the stand while filtering out the background noise of the large crowd of attendees speaking over their coffees.  Another issue was various kinds of asymmetries: at some times there were many in-person attendees interacting in front of the hybrid table but not many on the online side; at other times, remote attendees joined the hybrid table and talked with other remote attendees but there were no people there in person.  Furthermore, the in-person interactions were (perhaps inevitably) more natural, which sometimes led them to talk with each other exclusively and virtual people felt left out.  So while there were some benefits to the hybrid table, there were also issues, and in the end it may have been more successful at making in-person attendees feel they could interact with virtual attendees but not so much the other way around.

Experiments with Ask Me Anything and PLTea social events were successful. Talia Ringer organized a number of Ask Me Anything sessions; these were well-attended both virtually and in person.  It was exciting to see so much in-person interest since many of the people being interviewed were remote. Aviral Goel organized PLTea social events at every break in the conference schedule; these were also well-attended and appreciated. Some of these scheduled PLTeas allowed hybrid participation via our hybrid table set up described earlier. Events like this, as well as new kinds of hybrid social event experiments, should continue in future hybrid conferences.

Hybrid AV with Mirroring and YouTube Streaming may be Unsustainable for Volunteers. SIGPLAN’s flagship conferences have videoed their main sessions for several years now, which is already a lot of work.  Virtual conferences in 2020 increased that burden substantially, and supporting a hybrid conference with mirroring and YouTube streaming was simply an enormous load. In particular, having to intermingle prerecorded and live content created substantial scheduling and synchronization woes.  Despite some thorough training sessions and recorded videos, the in-person student volunteers faced several challenges for hybridization e.g., collecting all necessary information about a session, reminding the session chairs and others to follow the steps, moving virtually between platforms (Discord, Zoom) and physically between podium and A/V box, controlling the A/V box for right sound and video I/O in real time, and fixing glitches. Running all those consistently required several physical training sessions for the SVs. We also figured out the need for some floater SVs who monitored moving around the sessions. Another big effort was in the backroom—a team of at least six people always monitored whether everything was in place, kept an eye on the video stream (Youtube) and virtual part (Zoom), nudged session SVs about their tasks, and answered floaters.

We were amazed by the dedication of the student volunteers, led by Benjamin Chung and Leif Andersen, who made this work.  However, there was a widespread feeling among the organizers that this is an unsustainable task for volunteers to take on, at least based on then-available hardware and software support.  In addition, the complexity of the setup resulted in glitches that caused significant complications and stress for volunteers, organizers, and session chairs alike, and diminished the experience for attendees.  An additional challenge was that free streaming to YouTube diminished the perceived value of paying the conference registration fee for some attendees.  We think alternatives such as simply using Zoom without mirroring could however make hybrid conferences work at a reasonable volunteer effort level, with fewer glitches and helping to ensure that virtual registrants are getting something for their money.  Alternatively, we could try to continue the current support level, but hire professional A/V staff, but unfortunately this is likely to be too expensive based on current ACM conference budgets.

In a hybrid setting, make larger events accessible to all attendees, but focus on the in-person experience or the virtual experience, depending on attendance.  Unfortunately large workshops and conferences that were mostly made up of presentations tended to have lopsided attendance—and trying to make the event “natively hybrid” didn’t work so well: it was a lot of work, there was significant inconvenience to whatever part was in the majority, and the benefits to the minority weren’t commensurate.  Our suggestion for the future is, if a larger event is mostly in-person, focus on the in-person aspect of the event, but allow remote participants to see a stream and some way to ask questions (ideally via Zoom for the interactivity).  But if the larger event is mostly virtual—this happened with some European-focused colocated conferences and workshops, since Europeans mostly couldn’t fly to the US in October 2021—make it natively virtual, and orient the in-person attendees as a watch party (who can of course still participate).

Hybrid sessions should have in-person chairs.  Several people who chaired hybrid sessions remotely at SPLASH noted that this was difficult and awkward; future hybrid conferences should try to have in-person chairs whenever there is a true hybrid session (i.e. excepting only virtual-first sessions).  At SPLASH 2021 we tried but didn’t have much of a choice about this, as the in-person attendees were severely limited by COVID-19, but hopefully future conferences will be better able to follow this guideline.

Running poster sessions and the Student Research Competition both in-person and online was successful; adding hybrid elements was less so. People presenting posters or participating in the Student Research Competition (SRC) could attend an in-person session in the conference hallway if attending in person; both in-person and remote attendees were also invited to present remotely.  All judging for the SRC was done remotely, partly because the judges were remote and partly to ensure fairness.  A lot of people came to the in-person session which made it a crowded one; so much so that at times listening to the next person was challenging (of course, this is true at many purely in-person events).  In-person attendees really appreciated the chance for face-to-face interaction, while remote attendees valued the ability to participate even though they could not be in person.  We tried some hybrid elements: for example, our volunteers streamed some video of the session to the Discord video room, so that online participants could watch. However, it was very difficult for the online participants to interact in such a crowded session; doing posters separately seemed more successful. 

Q&A should be with the talk, not at the end. In the post-conference survey, a majority of people were not a fan of having a question and answer session after all talks in a session.  We put Q&A at the end mostly to simplify logistics for sessions that included both in-person and remote speakers, but in an approach where sessions are all-in-person or all-remote and remote participants are linked in via Zoom, it should be reasonably easy to put Q&A with talks.

Make remote presentations more interactive. For remote speakers, we played prerecorded videos to make the video quality better—and it probably was.  But if we had to do it again, we’d have them present via Zoom as it’s more dynamic and interesting to listeners. We highly recommend to have pre-recorded videos as a backup for the remotely delivered talkies with agreed stop points (e.g. every five minutes) to be able to have the chair resume the video with minimal interruption to the conference flow.  A related factor is that many speakers recorded video with relatively small videos of their faces; making those videos larger helps to make the video more compelling—even if it uses up screen space that could be used to present technical material, it’s likely a win in most cases. Also, we’d take Q&A via Zoom for similar reasons.  The sacrifice in quality would be more than made up for by the benefit for interactivity. The results from the conference survey back this up; respondents expressed a preference for Zoom Q&A over the Discord Q&A mechanism we used.

Mirroring was highly valued by some participants but also presented logistical challenges, and mirrored sessions were often poorly attended. We did mirroring for both Virtual SPLASH 2020 and Hybrid SPLASH 2021.  Mirroring was quite a lot of work in both years.  It required authors to prepare a video version of their talk, attend both their regular talk session and the mirror session.  It also required finding twice the number of session chairs.  In the purely virtual setting of SPLASH 2020, many of us felt mirroring was a win: authors got to present their work and answer questions about it to people all over the world, and for OOPSLA the schedule was coordinated well enough that authors could be awake for both mirrored sessions (this was an issue for some other venues, but that was more a coordination failure than a design failure).  Although the CS audience is concentrated in US/European time zones, there were enough people attending from around the world to make 12-hour mirroring work.  But SPLASH 2021 was different in two ways.  The audience was more concentrated in the in-person time zone, and the mirroring was only 8 hours later which meant a smaller part of the world benefited from it.  As a result, the mirrored sessions were attended very sparsely; in some cases only the session chair and authors were present.  In addition, the logistics of supporting both mirroring and in-person presentation made conference organization very challenging.  Better software support to reduce the logistical costs and finding some way to increase attendance in the mirror sessions are important for making mirroring work in a hybrid setting.

Some might be tempted to give up on mirroring entirely, but the post-conference survey shows that a substantial portion of the community values it. 35% of respondents said that mirroring is extremely important to them; another 15% support mirroring but have reservations; 40% have no strong opinion, and 11% felt some other format would be much better.  So there is a bit of a contradiction: few people attend mirrored sessions, but many people feel it is important to have them.  We should continue to think about creative ways to address this issue.  One possibility is to consider whether mirrored sessions would be better attended at SPLASH 2022 (where many people in the SPLASH community will be in North America, halfway around the world from the New Zealand venue) or if there are other ways to provide interactivity to remote participants, such as via workshops held in different time zones like those at POPL 2022.

Collecting talk videos and providing captions, however, was valuable.  Even if very few people attended the mirror sessions, many people appreciated having talks available on YouTube.  A number of people appreciated having captions on the videos—not only do they make talks accessible, but they can compensate for a wide range of other communication issues such as presenters who might talk too fast.  Although preparing videos and captions does create a bit more work for authors, it also means their efforts continue to pay off past the conference and are accessible to more people; we think this is something the community should keep from the virtual conference era. We did not see authors reporting many issues in creating the captions. Our approach was to provide step-by-step instructions for creating captions automatically using YouTube Studio, which is free and surprisingly accurate. It also lets the authors fix the errors and add punctuation. The authors could also use other services and send us the captions file (.srt). We had issues playing some of the captions in live streaming, mostly because of pulling the right caption file for a video. However, all the talks uploaded to SIGPLAN YouTube channel had the captions which were much appreciated. 

Conclusion

We should keep doing Hybrid conferences.  Although we may hope that COVID-19 will gradually become less of a concern for attending conferences in person, there have always been people who cannot attend in person but would still like to participate in various ways.  This is especially true for prospective authors who may not have the funding to travel, as well as those who may not be able to travel due to disability or to other commitments.  These members of our community have seen major benefits from virtual and hybrid conference offerings over the last two years.  Not everything we did at SPLASH 2021 was a net win, but as outlined above, we believe there are sustainable ways to offer Hybrid conferences that maintain these benefits while also offering the benefits of in-person conferences for those who can do so.

Bio:

• Jonathan Aldrich is the hybridization committee co-chair and the GC of SPLASH 2015.
• Sumon Biswas is the accessibility chair for SPLASH 2020 and 2021.
• Steve Blackburn is a strong proponent of time zone inclusivity highlighting those PL researchers outside of the EU and the USA time zones and is the GC for PLDI 2023.
• Benjamin Chung is the master video publisher and editor for SPLASH 2020 and 2021.
• Youyou Cong is the hybridization committee co-chair for SPLASH 2021 and 2022.
• Alex Potanin is the hybridization committee member for SPLASH 2020 and 2021 and is the GC for SPLASH 2022.
• Hridesh Rajan is the GC of both SPLASH 2020 and 2021.
• Talia Ringer is the hybridisation committee co-chair for SPLASH 2021.

Disclaimer: These posts are written by individual contributors to share their thoughts on the SIGPLAN blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGPLAN or its parent organization, ACM.