4 min read · Apr 3, 2023
Evaluation techniques are methods that are used to analyze the quality, effectiveness, and accuracy of any product/service/process. Almost all of us have been a part of these evaluation techniques in some way or the other, whether it be reviews, interviews, discussions, or surveys of any product, service, or process.
Focusing on the theme of evaluation techniques, we will be discussing two research papers. The first one is “How to conduct a heuristic evaluation by Jakob Neilson” and the second is “Usability evaluation methods: Mind the gaps by Estelle de Kock School, Judy van Biljon, Marco Pretorius”.
Going over the first research paper discusses heuristic evaluation which is a method for finding the usability problems in a user interface design, so that they may be catered to in the iterative design part. A small group of evaluators examine the interface and point out the usability problems that they find out, so it is difficult for a person to do a heuristic evaluation because they will not be able to see all of the problems present.
Each evaluator inspects the interface alone, once their evaluations are done they are then allowed to communicate with other evaluators, thus removing the opinions of others on the evaluation and this aspect is very crucial. These evaluations may be made into reports or communicated to an observer; written reports require additional effort from the evaluator whereas an observer reduces the workload.
Now the question arises what is the difference between this heuristic evaluation and our traditional user testing? In our traditional user testing the observer has to interpret the user’s actions as to how they are related to the design interface, whereas in heuristic evaluation the observer only needs to record the evaluator’s comments. So heuristic evaluation is for identifying typical usability issues and providing a generalized solution, whereas user testing focuses on significant issues and addresses them with specific solutions.
The evaluator analyses the interface multiple times and compares it with a list of recognized usability principles, this list is shown in the figure above. These heuristics are generalized rules that are common in a majority of user interfaces, in addition to this list the evaluator is also allowed to use additional usability principles.
Once the evaluators have completed their heuristic evaluations, their output is a list of usability problems regarding what exactly they found to be wrong.
So, this heuristic evaluation along with user testing can be applied in Pakistan as it is not such a troublesome task and I think people would be more comfortable using this as it divides the workload.
Overall, the research paper was not that interesting, and in my opinion heuristic evaluation could’ve been explained better, and also there were very few examples included, moreover I had difficulty understanding how they are determining the number of evaluators.
The second research article is quite similar to the paper that we just discussed, it discusses the application of heuristic evaluation and empirical user testing supported by eye tracking, to the website of a learning management system, to find out the difference between the information gathered.
I’ll not be going over evaluation techniques, heuristic evaluation, and user testing as that would become redundant instead, I’ll be directly going toward the research part for their differences.
Eye tracking, which will be used to support making comparisons between the two evaluation techniques, is tracking the eye when tasks are being performed to gain knowledge about cognitive operations. Eye tracking has been used to evaluate website designs, which has proved to help improve design interfaces. So, in this study eye tracking visualizations are used to compare user testing and heuristic evaluation.
An experiment was conducted in which participants were tested one at a time, they were briefed about the website, and the task list was explained after which the test was conducted, in which they were asked if they disliked anything if they couldn’t find anything, and questions like so, and along with an additional questionnaire at the end. Moreover, data collection was done in various formats, such as video recordings and participants’ facial movements along with keyboard and mouse movements.
The results showed that the evaluator and subjectivity are core components of both, moreover, it was also seen that heuristic evaluation was based on heuristic guidelines and evaluator interpretation, whereas user testing evaluation was based on direct assessments of participants’ system performance. User Testing focused on the activities completed, whereas heuristic evaluation took a broader perspective that extended beyond the tasks given.
The heuristic evaluation came out to be more extensive at a meta level, where as user testing evaluation was better at surface level and page level problems. HE did provide more problems than UTE, moreover, UTE also contradicted the post questionnaire. It was advised to think carefully before accepting questionnaire results that were not substantiated by objective measures or linked to any interaction theory.
So the research study gives us a good and in-depth overview of HE and UTE and their comparison made through eye tracking on websites. The study was quite interesting and it showed that our evaluation techniques can change based on our purpose and that there can be differences for different purposes.
