User:Legoktm/Blog/CVE-2021-33038

Note: No private messages were actually exposed to non-NDA individuals.

Wikimedia recently finished its migration from the old and outdated Mailman version 2 software to the newer and more modern Mailman version 3 software for our public and private mailing lists hosted at lists.wikimedia.org. We'll have more to write about this migration but for now I want to discuss and explain a security issue we discovered in the migration process: CVE-2021-33038: private list archives would be public for the duration of the import, which could take anywhere from seconds to hours.

Discovery
We have around 800 mailing lists and figured out how to migrate them individually from Mailman2 to Mailman3. We asked for some early adopters with list owners who were willing to be guinea pigs while we ironed out some of the initial bugs. The wikimediacz-l mailing list volunteered to be one of our early migrants and was the first non-test private mailing list that kept archives we moved over. We try to do all of our server actions in a transparent manner, so Amir announced in our public #wikimedia-operations IRC channel (then on Freenode, now on Libera Chat) that he was moving some mailing lists over, so people opened up the web interface to take a look. Majavah, a trusted volunteer, noticed that the wikimediacz-l archives were public for a few minutes but private after the import concluded. Once we confirmed this was an issue, we opened a private security bug report in our Phabricator installation: T281402 (now public). At this point we paused the migration of private lists with archives until we could fix the issue.

One question worth asking is, "Why we didn't notice this in our earlier testing?" We had set up a temporary test Mailman3 install called "lists-next.wikimedia.org", and practiced the migration process there, including with a private test list. But there are two primary reasons we didn't identify this as an issue earlier. First, both Amir and myself, who did most of the testing, have "superuser" accounts in the Mailman web interface, which means we bypass all permission restrictions, so we wouldn't see any permission denied error if the list was correctly set as private. Second, the test list we imported had only about 10 emails, which took about a second to import - not enough time for us to check that it stayed private for the duration of the import.

Identifying the cause
We are not running the latest versions of the Mailman3 software (we use Debian packages), so I wanted to be sure that this was still an issue in the latest version before we reported it as a security issue to the Mailman developers. I created a 2GB dummy mbox archive and then ran through the migration process on our Mailman3 test server (hosted in Cloud VPS) carefully watching the database state throughout the process.