As a developer, I have noticed that I’m not very good at testing apps, compared to having the app users test them. No matter how carefully you test your app, there’s nothing like the real-world test of having other people use it. Thus, making apps available to specific people for testing is very important to developers and product managers.
Beta testing is the second phase of software testing in which users (early users), who are not involved in the development of the software, use the software. The experiences of these selected users are then relayed to the developers, who make changes before the final release.
Beta distribution tools enable you make your app available strictly to specified people, so that you can have testers try it out and collect feedback before you offer it to the public.
TestFlight Beta testing is Apple’s attempt to provide a seamless way to distribute pre-release builds of iOS, watchOS, and tvOS applications to testers. With TestFlight Beta testing, you can invite up to 2000 testers using their email addresses. It’s free, easy to use and optional.
TestFlight is awesome but it isn’t without its shortcomings. One that easily comes to mind is the mandatory review process for external testers. This has created room for other beta distribution tools that can provide more features and/or bettered existing features in TestFlight . Most often, depending on specific needs, iOS developers and product managers have had to look beyond TestFlight when selecting a beta distribution tool.
In this write up, I will try to discuss and compare some top beta distribution tools available for iOS.
TestFlight Beta Testing
Apple continuously improves TestFlight. Considering that it’s owned by Apple, TestFlight is naturally the first stop when considering beta distribution tools for iOS.
- Ease of use.
- No dependency required.
- Easily add testers using email addreses.
- Large pool of testers. Currently 2000 testers. It’s free.
- App review for external beta testing. Apple review apps submitted for beta testing and must approve it before it can get to your testers. Though this is for initial submission and reviews now take few days, this is still a huge factor to consider when deciding on a beta distribution tool to use, because there will be occasions when you want to quickly get the app to the tester with the delay of waiting for review.
- Crash reporting. TestFlight’s crash reports are lagging behind those of third party services. It’s less intuitive and not as visual as other tools. It relies on users sharing their crash reports with Apple. This limits the size of crash reports received as users are more inclined to ignore crashes. I’m guilty of that :)
- Only one active build at a time. Previous builds can not be installed once there is a new build. Limited external builds per day. Six external builds per day seems enough but there are those days when you may require more than that number.
- Testers are required to install the TestFlight app.
- No seamless way to restrict build/update to a group of beta testers.
- Quite a number of forms to fill and questions to answer when submitting build.
- Support for only iOS. There are those occasional situations where invites are not sent to testers.
This seem to be everyone’s favorite. HockeyApp manages your betas and collects live crash reports for iOS, Android, Windows phone and OS X apps.
- Cross platform. This is a very import factor to consider when your app is available across multiple platforms. It’s a lot more convenient to have all apps in one place.
- User and Device Registration Management. This gives you good control of beta distribution; you can restrict access to certain groups. This is possible because you can group testers. If you want to recruit testers, HockeyApp allows you to build a recruitment page for inviting testers.
- Desktop App for easy distribution.
- Tester can install app from mobile website. It’s not necessary to install the client app.
- Good crash analytics.
- Integration with third party apps using webhooks. For instance JIRA webhooks allow your JIRA to indicate that an issue has changed, so HockeyApp can update the status of the corresponding crash group automatically. Note that this requires JIRA 5.2 or newer.
- Open source (Hockeykit).
- No review required.
- It’s not free. Though HockeyApp has different payment packages, the cost is still a factor for small development teams. There’s a free plan that has a maximum capacity of two apps. All packages include unlimited storage, crashes, users, feedback, symbolication and customer support. The tables below show the paid packages available:
- Setup for client apps (iOS to be specific) can be optimized further. Users have to register for the platform and then register their device. Also, after installation the user has to go to settings and accept to use the app.
- The user interface for app management is not as direct as it could be.
- The feedback page has poor UI.
- Testers are not notified when feedback is replied.
- Third party dependency integration.
My personal favorite. Fabric is a platform from Twitter featuring a variety of tools that assist with testing, gathering, crash reporting, understanding users and growing business.
- Great crash log. Fabric owns Crashlytics. Crashlytics is arguably the best tool for comprehensive crash reporting. A very good framework for analytic and real time data collection.
- Cross platform support for iOS and Android.
- Easy setup. Integrating fabric into your app could not be easier. Oh it could be. Buddybuild has proven that. Fabric desktop app could be used to install the rest of the tool awesome tools that comes with Fabric.
- It’s free! Fabric as a tool offers a lot and it all comes for free.
- Easy deployment. The Fabric desktop app detects archive when you are done archiving your app on Xcode. Few clicks and the build is deployed. A lot of useful metrics and tracking of custom events.
- Distribute and maintain multiple versions at once.
- Ability to scope errors and bugs by users and devices.
- Good customer support.
- Multiple frameworks in one package.
- No review required.
- Web interface doesn’t have much useful information.
- Doesn’t allow build to be sent to same tester again. I have had situations where testers deleted a build or for some reason the tester requires the build to sent again. The work around I use, is to build again or remove tester access to that build and deploy again.
- Mobile app is a bit buggy.
- Dependency is involved.
The TestFairy homepage says “Android and iOS Mobile Beta Testing Has Never Been Easier”. That could not be more true. The most interesting feature, to me, is it provides a video of exactly what the testers did that led to a crash or bug.
- Video record of tester actions. Bug or crash comes with video playback to enable developers reproduce the bug or crash.
- Robust data logging. Monitors memory, CPU, GPS, and battery.
- Shake to give feedback. You shake the device and enter your feedback. Interesting.
- Very good crash report.
- Simple and easy to setup and use, yet powerful. You upload IPA, then follow the doc to integrate the SDK.
- Cross platform. Native and hybrid.
- Support for enterprise IPA’s.
- Easy integration with third party apps. You can easily integrate apps like Slack, HipChat, through
- Webhook on account preference page.
- Uploading apps via build server; Circle-CI,
- Jenkins and other continuous integration platforms.
- TestFairy has a community page.
- No review required.
- As with every alternative to TestFlight, dependency is involved.
It’s not free. Payment plans are shown in the table below:
The new upload feature sometimes fails with no information as to what went wrong.
- The dashboard UI could be better.
Certainly used by respected brands, buddybuild is very easy to setup (until creating a build on the website fails and you see no good intuitive action to take other than change some variables and try again) but powerful beta distribution tool. Almost every setup process (including technical setup) is automated and can be done on the dashboard. A non-technical person can follow the doc and set up buddybuild for an app. To be blunt, buddybuild works like a “charm.”
- Instant replay. Reproducing bugs and crashes is so easy as each bug and crash report comes with a video replay.
- Easy to set up. From buddybuild dashboard you can upload your app on github, gitlab or bitbucket. buddybuild does all the heavy work. You then upload your provisioning profiles and start inviting testers. When you are ready to add the buddy SDK, you only have to click buttons.
- Pinpoint crash report.
- Awesome feedback reporter and visual bug reporter.
- Monitors your app git repository. A new build will automatically be kicked off every time a 'git push' happens to your repository.
- Easy integration with third party apps. For instance, buddybuild can be configured to automatically open a JIRA issue for each piece of feedback and crash report you receive on your app. The process to achieve this is well documented on buddybuild doc.
- Cross platform. Supports iOS and Android.
- Awesome UI.
- Run unit tests.
- As with other tools, dependency is involved. Building an app from repository is awesome and saves the developer some work, but granting access to source control system might not appeal to everyone.
- It’s not free. Payment plans are shown below:
AppBlade provides a platform that can manage mobile applications, users, and devices in one convenient tool. One feature I love is, you can install applications remotely, force updates, uninstall them.
- Remotely install applications.
- Infinitely customizable. You can customize a lot (color, logo and more).
- Cross platform. Support both Android and iOS.
- No client application to install. Testers can install app via web.
- No reviews.
- Integration with continuous integration e.g Jenkins.
- No limits.
- Ability to create custom release tracks to manage several app release.
- In-app feedback.
- Good crash report.
Cons * Dashboard UI is poor. * Quite some work to do when integrating SDKs. * As with other tools, dependency is involved. * It’s not free, but is arguably the cheapest I have seen. The table below shows AppBlade’s payment plans. Note: First 25 devices are free of charge across all plans. All plans include access to live support, our open-source SDK, and no limit on users. Just pay for the devices you use.
Installrapp tries to make the entire process of beta distribution (setup, reviews and processing time) faster. With a few a clicks, my app was ready for distribution. It syncs with Apple developer portal to register your devices, update provisioning profiles, and update your IPA automatically.
- Installrapp automatically updates the
- Provisioning Profile with new devices, if you enter your developer email and password.
- Limit tester’s number of device.
- No client app to install. Tester can install build via email.
- Cross platform support. iOS and Android.
- Installation via mobile browser. Testers don’t have to install any client app.
- As with other tools, dependency is involved.
- It’s not free. The pricing plan is shown below
Deciding what tool to use
Bottom line, the essence of these tools is to help you deliver your app to testers and collect feedback. But when deciding on what tool to use, you have to look beyond beta testing. The services provided by these tools are also employed in commercially released apps. User standing, user behavior and other analytics can be used to grow businesses.
I have been in engagements where we used a combination of tools. Most times it is TestFlight and any of the alternatives. I can only advise that you look at what your needs are, and choose a tool that satisfies those needs at comparative cost and convenience.
I cannot predict the future of beta distribution tools. It is obvious though that a lot more effort will be put towards making tools more convenient and less work for developers, testers and project managers, collection and analysis of more information (bug and crash report, useful metrics).
There are other awesome tools that are not in this write up.
If you have a favorite you really want me to try out, please send me a message. You can also add your own review in the comments section.