On August 21, 2023, the IOT denylist will be updated to use a new weekly generation process and setof classifiers to identify hotspots that are gaming the IoT Proof of Coverage system.
On January 13, 2022, thecommunity approvedthat Nova Labs would be allowed to operate and enforce adeny list to handle widespread abuse of the Helium Proof ofCoverage (PoC) system.
The system that was put in place relied on both simple metrics and more involved learning algorithmsto detect various gaming techniques which would, with high confidence, flag a gaming hotspot.
The list of identified hotspots is placed on the deny listwhich, in the current system is picked up by the IoT Oracles to exclude those hotspots from earningPoC rewards.
Owners of hotspots on the deny list could issue removal requests to crowdspotwith evidence that they were mischaracterized and should be removed from the deny list.
The following are the limitations encountered using this approach as implemented currently:
- The algorithmic detection of bad actors has a small but noticeable false positive rate.
- Many deny list removal requests were submitted, often with multiple submissions per hotspot.
- A large set of removal requests attempted to create confusion by submitting false information withhotspots that were correctly added to the deny list.
- The number of removal requests quickly exceeded that which could be reasonably processed by a teamof volunteer reviewers.
- Maintaining and managing a well-staffed and motivated team of volunteer reviewers has proven verychallenging.
- The small number of false positives and a large percentage of false submitted information,combined with a slow, very manual review process, resulted in some hotspots remaining on the denylist for weeks or months before being reviewed.
The deny list system is a step towards the end goal of being able to augment the PoC pipeline withnormal, HIP-approved, and open-sourced classifiers that detect gaming techniques and scale rewardsdown appropriately as PoC data is processed.
Since the HIP process takes time and doesn’t allow for a fast reaction to new PoC gaming vectors,the deny list offers a way to apply fast stop-gap measures to stop gaming attempts. Once thosemeasures are proven out and solid, they should be promoted into the PoC pipeline using the normalHIP process.
To be effective the deny list must be authoritative in that an individual hotspot on the deny listcan not be appealed. What can be appealed is the algorithm used to put that hotspot on the list byanalyzing all tagged hotspots with a given algorithm and proving that it is flawed. This improvesthe algorithm itself, reducing false positives, while avoiding non-scalable human reviews.
The new process for managing the deny list must have the following properties:
- Detection criteria are transparent, public, and reproducible using data that are available toeveryone. The exact code may not be made public to reduce the breakdown speed of a detectionalgorithm.
- Removals from the list are automatic and happen quickly (within 7 days) when a new denylist isgenerated, assuming hotspot behavior has adjusted.
- Hotspots can be added manually to the deny list in an emergency measure but will be removed fromthe manual list automatically after 14 days. This allows for fast reaction while forcing thealgorithm to improve within the given window of time and does not let hotspots languish on thedeny list.
We recognize that any detection technique for bad actors on the IoT network will have an associatederror rate. The objective is to minimize the impact of that error while maximizing the number ofhotspots that are correctly added to the deny list, so that rewards may be distributed as fairly aspossible. As mentioned above, the approach to false positives is to improve the detection criteriato reduce the error rate.
The new deny list detection criteria is based on classifiers which have the following properties
- A classifier produces an 0-1 output as to the confidence of a hotspot providing proper coverageaccording to the classifier algorithm
- A (growing) number of classifiers are used to produce a scorecard for a hotspot on the deny listwhere each classifier score is weighted to produce the final score. This scorecard is tied tothe deny list version the hotspot is on and is produced and statically available.
- A manual classifier is used to allow for emergency blacklisting of hotspots. This classifieroverrides all other classifiers. Hotspots are automatically removed from the manual deny listafter 14 days.
- The classifiers are run every 7 days to produce a new deny list and may include improvements tothe previous classifiers as well as new classifiers to improve detection criteria and reduce falsepositives.
- The current deny list is replaced by a deny list that is generated by the initial set ofclassifiers. The target date for this is August 21, 2023
The following classifiers will be used for the initial deny list
Terrain-Aware Signal Verification
Beacons and data sent over the IOT network do not propagate through significant terrain, such asmountains, or hills. Signals are generally only received if there is a clear line of sight betweenthe transmitter and the receiver. We use the public NASA Shuttle Radar Topography Mission (SRTM)data (available at https://www2.jpl.nasa.gov/srtm/) to determinethe terrain profile between hotspots. This terrain profile is used to evaluate the line of sightbetween the asserted locations of hotspots and their witnesses. This concept is illustrated below:
The line of sight between the transmitter and the receiver is shown in orange, and the terrainbetween them is blue. In this example, the line of sight is blocked and successful radiocommunication is unlikely. We measure the amount of terrain blocking communication by taking thearea under the terrain curve and above the line of sight. This number will be zero for a perfectline of sight between the transmitter and the receiver, and large if it intersects significantterrain features. We call this measurement “terrain intersection”.
The terrain intersection is calculated for all the communication between a hotspot and its witnessesover days. A hotspot that has a high terrain intersection for most of its witnesses and receivedwitnesses is likely mis-asserted, or being rewarded for IOT coverage which it is not providing. Hereis an example of the terrain intersection for a hotspot with abnormal behavior, and one which iscorrectly asserted:
Note the line of sight in the abnormal case goes through a very large amount of terrain.
Note that the line of sight is well above the terrain in the normal case. The classifiers will allowsome terrain intersections to account for some terrain overlap.
A hotspot is added to the deny list when the terrain intersection for its received and transmittedwitnesses is unusually high compared to the other hotspots on the network. We currently define thethreshold as three standard deviations from the network average, but this can be adjusted as newdata becomes available.
Witness Distance Sensitivity Measurement
Hotspots that are close together tend to share the same witnesses. Distant hotspots may have nowitnesses in common with each other, due to terrain, obstacles, and the range limits of theirradios. There is a way to measure how similar two hotspots are by looking at the lists of theirwitnesses. This measurement is called theJaccard index. We also distinguish the analysisof inbound vs outbound witnesses for the hotspot. It works by counting the total number of witnessesbetween two hotspots and dividing by the number which is shared. Here is an example of thiscalculation for a normal hotspot:
The amount of shared information between this hotspot and its witnesses falls off naturally withdistance. Here is an example of an abnormal hotspot:
Witness similarity that does not decay with distance is a strong indicator of hotspots that areclose together, or relaying witnesses artificially, and are added to the deny list.
Witness Similarity Measurement
Note that in addition to similarity not varying with distance, in this instance many of the inboundand outbound similarity values for a given peer are very close, or identical, to each other. Ingeneral, the similarity is not expected to be so symmetric because of asymmetries in thetransmit/receive paths.
For example, a hotspot up on a hill with good antenna placement may see poorly placed hotspots belowit in a suburban environment, but those hotspots may not all see each other. A hotspot with ahigh-gain antenna will have more receive sensitivity than a hotspot with a stock antenna. A hotspotwith nearby interference may have worse receive sensitivity for distant signals, etc. Thus we wouldexpect, and we do observe, some variation in the inbound/outbound Jaccard index for a given peer.
Measuring the difference between these examples is accomplished by correlating the Jaccard index fora given witness to the distance between the points on the graph and seeing how many of thoseinbound/outbound pairs are similar to each other. Hotspots that are abnormally similar in this senseare added to the deny list.
Manual Classifier
As a backstop measure, hotspots can be added to the deny list manually. This classifier will alwaysadd any hotspots in the manually provided list, but will age out added hotspots after 14 days ofhaving been added unless removed manually.
