By John Passarelli – Senior Machine Learning Engineer

Introduction

FoxyAI’s artificial intelligence looks at property images and scores the condition and quality. This visual examination, while seemingly on the mind of every prospective buyer touring a house, is a missing piece in many Automated Valuation Models (AVM). At FoxyAI we believe that just as a home buyer scrupulously inspects the visual quality of a property, so should the machines. 

A prospective customer felt similarly that their AVM could use the visual insight from FoxyAI models and the results are detailed below.

What Are AVMs?

Have you browsed predicted sale prices at Realtor.com, Trulia, or Zillow? Maybe these prices made you ponder about buying/selling a property? These pricing models are consumer-facing examples of Automated Valuation Models. AVMs attempt to determine the value of a property automatically and at scale.

How to Measure Performance of AVMs?

Ultimately, the “value” of a property is determined at time of purchase, when the contracts are signed and there is a final sale price. This is the ground truth data. The goal of an AVM is to have the predicted sale price be as close to this final closing price as possible. To evaluate the performance of an AVM, AVMs are run using historic data and that output (the predicted sale price) is compared with the final closing price. 

Experiment with FoxyAI

This experiment was conducted by a prospective customer (spoiler alert: now current customer) to determine if FoxyAI models could boost their AVM performance. 

The customer’s AVM (like most) uses tabular style data such as tax price, number of bathrooms, square footage, etc. to predict the home value. These features do not include the critical visual piece – the quality of the house. 

Prospective Customer Goal:

Determine if the addition of FoxyAI Condition and FoxyAI Quality models improve their existing AVM. And, if there is improvement, measure the performance increase amount.

Data:

The prospective customer collected 19,105 available single family properties in Cuyahoga County, Ohio that sold between 2018-2022 (resulting in over 400k photos for FoxyAI Quality Score Model to score).

Test:

The prospective customer tested the performance of their AVM without FoxyAI Quality Score results and with the FoxyAI Quality Score results which we will name: Method #1 “Foxless” AVM and Method #2 FoxyAI Enhanced AVM.

80% of the properties were used as training data, and the remaining 20% of the data was used to test the model. 

Method #1, the “Foxless” AVM, utilizes an XGBoost model with 46 tabular features previously determined as important base features for an AVM. Method #2, FoxyAI Enhanced AVM, is exactly the same as the first AVM but includes the FoxyAI Quality Score as a feature.

Results:

Below are a variety of metrics provided by the prospective customer comparing the AVM with and without FoxyAI. Beneath that we will go into more detail understanding these metrics.

Method #1 – ”Foxless” AVM

Method #1 – FoxyAI Enhanced AVM

MAE – Mean Absolute Error

PPE10 – Percent Predicted Error within 10 % (standard AVM metric)

Feature Importance

Interpretation: A standard metric of measuring AVM accuracy, PPE10, is improved by 5% by including the FoxyAI Quality Score as a feature (Note: In Data Science, the use of the word “feature” refers to a model input).

An advantage of using gradient boosting is one can easily retrieve the importance score for each attribute (as described in this link). There are 47 tabular attributes (such as tax price, number of bathrooms, square footage, etc) for each property that the model utilizes to predict the price – one of which is the FoxyAI Quality Score model. The FoxyAI Quality Score was determined to be the third most important feature. It is time for the machines to incorporate visual property intelligence and FoxyAI is at the forefront.