Performance and Accuracy of TensorView
In order to properly evaluate the performance of TensorView on our example survey, we have employed three metrics: total work time, false positive rate, and false negative rate. In this case, total work time refers to time spent conducting the survey itself - preparing the survey shapefile, running the tool, and performing the manual verification, field-surveying, and research steps. It does not include the development of the tool or the neural network training set, as these can be considered sunk costs in realistic settings - it is far more likely that a practitioner will utilize an existing neural network rather than create a new one. The false positive rate refers to the number of images erroneously identified by the tool as containing abandoned buildings when they cannot conclusively be determined not to by human inspection. The false negative rate is the inverse of this - it refers to the number of images that the tool identifies as not containing abandoned buildings when they, in fact, do.
Total Work Time
The overall example survey, from initial shapefile development to the completion of the archival research, required approximately 5 hours of time. Of these, 2 hours were spent performing computation, and the remaining 3 required human input. Of these, 1 1/2 hours were spent performing the field survey, which consisted of little more than driving to each location, photographing the site in question, and moving on. This is in contrast to manual surveying - we estimated that it would require approximately 5-6 hours of walking and driving in order to adequately examine the areas covered by the study. It is also notable that the 1 1/2 hours of the field survey were the only portion of the study that were not spent "deskside"; this is opposed to the extensive fieldwork required for a full manual survey. Though both field-survey time and computation time scale with the size of the survey area, the relationship has rather a slower growth rate than for manual surveying. Based on the time required for the sub-area survey, a full automated survey of Guelph would be expected to require approximately 92 1/2 hours - a period equivalent to approximately 4 days of non-stop computing. Factoring in the other steps of the study, we anticipate that a full end-to-end study of Guelph using this workflow would require approximately 6-7 days, of which 3 would require human labour. This is in comparison to a walking survey which, to cover the entirety of the City of Guelph, would likely require many weeks of work, significant transportation expenditures, and be subject to inclement weather and external factors.
Due to budget constraints, the processing time was much longer than would be encountered in a typical GIS environment, as this study utilized a decade-old consumer desktop computer as its primary source of computational power. Because high-power workstations are common in GIS practice, it can be expected that processing time would decrease dramatically in these contexts. Additionally, as TensorView is still in the proof-of-concept phase, many of the manual steps could be automated or streamlined in further implementations, decreasing the manual workload by a considerable amount - most notably, the shapefile preparation steps can be automated, and the manual review step could be significantly streamlined.
False Positives and False Negatives
Of the 3,358 images examined by TensorView during this study, 433 were identified as containing abandoned buildings - 12.9% of the total. Of these 433, 58 images could not conclusively be determined as abandoned or not, requiring field surveying - 1.7% of the total. The remainder - 375 images, 11.2% of the total - constituted false positives of various types. The 2,925 images identified as not containing abandoned buildings did not include any false negatives.
False positives can, very generally, be divided into two types: egregious false positives and ambiguous false positives. An egregious false positive occurs when an image is classified as abandoned that does not contain a building at all, or can very clearly be seen not to contain any abandoned buildings. This type of false positive typically indicates that there are weaknesses in the training set - either a new category is needed, or a certain category is "weak" and must have more images added to it in order to improve the neural network's understanding of what features are important. An ambiguous false positive is more difficult, if not impossible, to eliminate outright. This type of false positive occurs when the classifier identifies a building that could be abandoned, and requires human scrutiny of the image in order to determine its true status. Of the 375 false positives, 167 were egregious and 208 were ambiguous.
Though clearly capable of detecting features of interest, TensorView suffers from many of the same limitations as other remote-sensing technologies. Most notably, features of interest can be obscured while the Street View car is driving by - whether by permanent features like vegetation, or simply by large vehicles (busses, trucks, etc.) in neighbouring lanes of traffic. This problem is exacerbated by Street View's low temporal resolution, as, though it is unlikely for a location to be obscured in two separate images, it may be months or years before that location is imaged again.
Additionally, TensorView is somewhat \wasteful" with network resources due to the structure of the Google Street View API. At present, the API does not provide any means of retrieving historical imagery, or of determining what panoramas are within a given area. This requires a "guess and check" search for panoramas, which consumes time and bandwidth that could otherwise be used for other tasks.