Reliable CI systems with automated testing saves Mozilla’s Firefox iOS team hours of engineering time
"Working on open source projects forces you to be both diligent and transparent about the changes you and your team are making, while allowing the community to review, iterate, and help to improve your work," says Emily Toop, Mobile Software Engineer at Mozilla.
Toop and the rest of the Mozilla mobile team build many projects — including the iOS version of the Firefox browser — in the open. However, maintaining the integrity of the codebase for repos like Firefox, which has over 1,350 forks, and approximately 5,500 commits by 75+ different contributors, often requires in-depth pull request reviews, proper testing and code coverage. All of which require a complete continuous integration (CI) system.
Unfortunately, setting up a reliable CI system to support their open source projects proved to be both a time-consuming and frustrating process for the Mozilla team.
Frustrating because the mobile team wanted to focus on building their apps rather than maintaining their build infrastructure. And problematic because flakey testing infrastructure slowed their release cycles, and degraded the quality of their codebase.
No matter which CI service they tried, it seemed as if a member of the team was always on-call to ensure their build, deploy and testing processes were, in fact, working.
"First we tried Xcode server, which seemed to break with each Xcode update. Then, we turned to Travis CI but we quickly found ourselves in the same situation: constant breaks with little insight into the cause," recalls Toop. "If we weren't constantly maintaining these solutions, they seemed to fail endlessly."
Toop and the team tried CircleCI to no avail, and she personally spent three months focusing her efforts on scripting a development pipeline with Fastlane.
After using Fastlane to set up a development pipeline she felt confident in, Toop shifted her efforts to work on the Firefox app itself. But almost as soon as she moved her focus away from their Fastlane-powered build pipeline, it crumbled as well.
"Regardless of the number of contributors on the project, the existing CI tools failed to provide us with a reliable and manageable build system," says Toop. "It seemed like no matter what we tried, we'd always run into the same problem: the tools we needed to build the Firefox app required endless maintenance."
After months of trialing different solutions, the Mozilla team gave up on finding a CI provider that worked for their team and ultimately reverted back to manually testing code changes.
Shortly thereafter, Farhan Patel joined the Mozilla mobile team, and was responsible for reviewing the majority of incoming pull requests for the Firefox app. Tasked with combing through hundreds of pull requests, he felt the pain of not having a CI solution in place.
"I would try to run our tests on Firefox's open pull requests before merging them into master," explains Patel, "but without a CI, this was an extremely time-consuming process."
Growing frustrated, he started asking his network for CI recommendations and was introduced to buddybuild. Thinking anything would be better than their largely manual setup, he decided to give it a try.
"After the team's previous experiences, it seemed like it would be a lot of work to set up and manage a CI solution. Buddybuild is the first solution we've seen that easily manages our tests, dependencies, and different permission settings within the Firefox app," says Toop.
In reality, Patel says set up "wasn't more than a day's worth of work, and anything that wasn't supported, the buddybuild team was happy to add for us on the fly."
What surprised Patel most was that he "didn't have to make any code changes, or explain our dependency managers. Buddybuild was able to scan the project and correctly infer the configurations needed to properly build our app."
"Buddybuild correctly set up a build environment for us out of the box and it was super straightforward to further customize our workflow," says Patel.
Running tests reliably was a core requirement of any CI system for the Mozilla team. "Tests give us a way to screen pull requests for bugs before they're merged into our codebase," says Patel. "With so many contributors it's important to maintain the integrity of our codebase, and tests give us a sure-fire way to do that."
"We've always had tests for the Firefox app, but weren't able to run them consistently before buddybuild — they were prone to timing out, or would take an unreasonably long time," says Toop.
"Now with buddybuild, we have a reliable framework that runs tests and lint on every pull request before it can be merged," explains Patel. "It saves us from combing through each commit to make sure there aren't any mismatched semicolons or extra spaces."
"I'd say buddybuild saves us an hour of review time for each pull request. Time we now put towards building new features and improving our code coverage," says Patel.
Mozilla's favorite buddybuild features
Pull Request Support Auto-build open pull requests to ensure new changes are safe to merge
Instant Deployment No processing time or review processes needed
Annotated Screenshots Testers send annotated screenshots of your app with their comments
Today, the Mozilla Firefox team uses buddybuild for all of their mobile projects.
Toop has just onboarded Mozilla's newest app, prox, with buddybuild. For Toop, the biggest time saver has come from completely automating the build and deployment process. "I've been able to harness buddybuild's full suite of tools with this new project," says Toop. "I can plan the build, deploy, and beta testing cycles using buddybuild. As a team working on an open source project, we're able to move ahead with confidence knowing we have a solid toolset to rely on."
"We've set it up so it's impossible to commit to master unless all tests are passing. Buddybuild's given us an easy way to ensure the quality of the app's codebase stays up to our standard."
For Toop, "it's been super useful to have integrated Slack notifications to let everyone know if there's a build failure, or if there's a new build available to download. I'm also thoroughly enjoying buddybuild's SDK because each bug report gets filed in buddybuild, GitHub Issues, and as a new Trello card."
"Our bugs aren't lost," says Toop, "and for the first time, I've found a CI that impresses me instead of constantly breaking."