Identifying Real Estate Development Opportunities: Web-Scraping, Regex Patterns & String-Searching Algorithms

Abstract

Web-scraping and data mining algorithms are used extensively by hedge funds, equities traders, digital marketers and in the technology sector more broadly. Contrastingly, the real estate development industry continues to use traditional, manual methods to identify and pursue new development opportunities with the exception of mapping software which has been widely adopted. The lack of adoption of these technologies is primarily due to the difficulty in identifying, retrieving and processing the required data rather than an inherent lack of data. To the contrary, there is a wealth of public and private information available to the real estate development industry that can provide value if collected and analyzed efficiently and at scale using algorithms. To test this hypothesis, the author has built a functioning web-scraping and data collection platform that demonstrates how large amounts of data can be retrieved and processed at scale. This thesis evaluates the effectiveness of using web-scraping algorithms to search for real estate development and land rezoning opportunities from publicly available local Government data. The focus area of the thesis is Sydney, Australia and the subject of the thesis is the Aiden1 platform that is owned by the Principal Investigator and author. The platform uses automated web-scraping algorithms to parse publicly available local Government data for keywords that indicate a prospective development opportunity or an instance of imminent land rezoning. The results of this research demonstrate the effectiveness of adopting web-scraping technologies and the usefulness to real estate development professionals.