Unit tests are a Software Engineer’s best friend no matter what language you are writing in. The purpose of Unit Tests is to validate and protect existing code, this in turn enables teams to move faster with less risk.
“Why do most developers fear to make continuous changes to their code? They are afraid they’ll break it! Why are they afraid they’ll break it? Because they don’t have tests.” — Robert C. Martin
But one topic that is not discussed as frequently, from my experience, is: how long should it take to run a Unit Test? It really isn’t something that a lot of people think about but over time can be come a problem that will have you regretting some of your early decisions.
Typically the response I get when I ask this question is each test should take anywhere from 0.01 seconds to 1 second max to run in isolation. As we all know, that isn’t always the case! I’ve even seen some that take 7.6 seconds to run…. Yes, I’m not lying.
Overall the general rule of thumb is: the faster they are, the more likely Software Engineers are to run them. This could not be more accurate!
If you use a CI provider, this can be a real dollar cost
Scaling becomes very difficult
Impossible to do Test Driven Development
Increased feedback loops
Let's say you have a test suite that consists of 1,000 Unit Tests. Within that suite 90% of tests complete in 0.01 seconds each, and 10% complete in 0.1 second each.
What’s the big deal right? Lets do some quick math…
(1000 * 0.9)(0.01 s) = 9 s
(1000 * 0.1)(0.1 s) = 10 s
It takes longer to run 10% of your tests than the remaining 90%!
This approach isn’t a scalable solution. So how do you solve this problem? How do you even realize it’s happening?
With regards to iOS Development, there are some approaches that help to reduce test run time.
Adopt MVVM and heavily unit test your ViewModel
Modularize
Avoid TestHost Apps
Properly mock all of your dependencies
Pay attention to the run times!
I personally was interested in how to solve this on a previous project I was working on, so I played around until I was able to gain better insight into our problem at the time.
This Python tool allows you to gain some understanding into how your tests are performing, and even set a threshold if you’d like. Perfect for running on your CI provider.
I made a sample project named TestSpeedAnalysis that had 200 Unit Tests. 100 of them had a random delay added in that varies between 0–3 seconds.
The script is configurable based on your needs and provides various options that can be passed.
-s, --strict — Will fail CI if tests are detected above threshold
-p, --path — Path to your xcodebuild logs from your test run
-m, --max — Custom defined threshold that is agreed upon by the team (defaults to 0.01s)
If you’re curious about how your tests perform, I would recommend checking out the tool!
As a seasoned software engineer with extensive experience in the field, I can unequivocally affirm the crucial role that unit tests play in the software development lifecycle. The notion that "Unit Tests are a Software Engineer’s best friend" resonates deeply with me, as I have witnessed firsthand the transformative impact they can have on code quality, development speed, and overall project success.
The quote by Robert C. Martin encapsulates a universal truth in software development – the fear of making continuous changes to code stems from the lack of adequate tests. This fear is not unfounded, considering the potential risks associated with untested code modifications. In my career, I've observed how robust unit tests act as a safeguard, empowering development teams to iterate swiftly with confidence.
The article rightly delves into a less-explored aspect of unit testing – the duration it takes for unit tests to run. The author highlights a crucial point: the speed of unit tests directly impacts a developer's willingness to run them frequently. I have encountered this scenario countless times, where lengthy test suites become a hindrance to agile development practices.
The provided mathematical example brilliantly illustrates the issue at hand. If a test suite comprises 1,000 unit tests, with 90% completing in 0.01 seconds and 10% taking 0.1 seconds, the overall test suite runtime becomes imbalanced. This not only affects developer productivity but also has cascading effects on continuous integration, scalability, and the feasibility of adopting test-driven development (TDD) practices.
The author then shifts the focus to solutions, particularly in the context of iOS development. The suggestions align with industry best practices, advocating for the adoption of Model-View-ViewModel (MVVM) architecture, modularization, avoidance of TestHost Apps, and thorough mocking of dependencies. These practices contribute to shorter test run times, enabling a more efficient development workflow.
The mention of the Python tool, "test-speed-analysis-ios," adds a practical dimension to the article. This tool, developed by the author, showcases a proactive approach to addressing the issue of slow-running tests. By analyzing xcodebuild logs and allowing developers to set custom thresholds, it provides valuable insights into test performance and aids in maintaining optimal run times.
In conclusion, this article not only highlights a critical yet overlooked aspect of unit testing but also provides tangible solutions backed by real-world experience. As someone deeply invested in the field of software engineering, I commend the author for shedding light on this essential topic and offering a tool that can benefit developers facing similar challenges.
The vast majority of unit tests should take at most a few milliseconds to run, and preferably in the range of microseconds. Why? Because unit tests should be verifying mostly isolated, small units of logic.
Running unit tests should be fast, the sooner an issue is found the quicker the turnaround time. On average, a unit test should take a few milliseconds to run. This will help ensure that tests are narrow and isolated.
The test automation usually takes 12 mins on average to run. They are UI tests that execute different user flows in the web app and assert confirmations to make sure the common flows are working fine.
By using mock objects, developers can isolate the code being tested and focus on the behavior of the unit being tested. This approach can also help avoid issues with external systems, such as network connectivity or performance.
Testing should be done during the whole development lifecycle. Once the whole technical project is developed, all the features should be tested as a whole unit. The types of tests depend on the type of your project.
A good rule of thumb is to test only the public interface and behavior of your unit, and not the internal implementation details. You should also avoid testing trivial or obvious code, such as getters and setters, unless they have some logic or side effects.
Michael Feathers, in Working Effectively With Legacy Code, on pages 13-14 mentions: A unit test that takes 1/10th of a second to run is a slow unit test... If [unit tests] don't run fast, they aren't unit tests.
Fast: Most unit tests should take only a few seconds each to run. If they're taking several minutes or longer, then they affect the team's productivity, and developers will be less likely to run them regularly and consequently. This ultimately reduces the value of the unit tests.
To run a usability test effectively will take between 30–60 minutes per participant. Of course, depending upon the complexity of what you're building, this length of time will vary, but in my experience, an hour is about the maximum time I'd recommend.
"When your testing needs grow to the point that you need to run multiple tests on different devices at the same time. You can run tests manually on different devices, as long as you have enough manual testers. However, if you want to test your system by running tests simultaneously, you need to automate your testing.
Unit tests test a single piece of code, while integration tests test modules of code to understand how they work individually and interact with each other. Unit tests are fast and easy to run because they “mock out” external dependencies.
Each unit would generally comprise 2-3 hours of the combination of in-class and self-study time per week on average, such that a 3-unit course would generally comprise 6-9 hours per week on average, or approximately 78-117 hours over 13 weeks.
It takes a lot of time because tests have to be done every time the code changes. It is hard to find and fix problems because it is tricky to test each part separately. Developers often do manual testing themselves to see if their code works correctly.
How Long Will It Take? An electrical safety check takes between 3-4 hours to complete. Nevertheless, this depends on the size of the property and the number of circuits that need testing.
Address: Suite 592 642 Pfannerstill Island, South Keila, LA 74970-3076
Phone: +9617721773649
Job: Marketing Producer
Hobby: Skydiving, Flag Football, Knitting, Running, Lego building, Hunting, Juggling
Introduction: My name is Tuan Roob DDS, I am a friendly, good, energetic, faithful, fantastic, gentle, enchanting person who loves writing and wants to share my knowledge and understanding with you.
We notice you're using an ad blocker
Without advertising income, we can't keep making this site awesome for you.
