News reports routinely describe the teacher labor market as “dire” and “in crisis.” In recent years, influential media outlets across the country have described “unprecedented hardships” and “critical shortages” because, they report, demand for qualified teachers far outpaces supply.
But school leaders and researchers know there’s more to the story. Teacher shortages are not universal and staffing challenges vary widely by subject area and school type. Yet public understanding of teacher shortages typically lacks these nuances, and policymakers often don’t have detailed enough information to address schools’ needs.
What if there were a low-cost way to get a close-to-real-time snapshot of the demand side of the teacher labor market? Fortunately, there is: web scraping, an automated data-extraction technique that regularly exports and refreshes data from the Internet. Once set up, web scraping can quickly build and maintain updated data sets primed for analysis—and action.
We used web scraping to create a comprehensive database of teacher job postings in Washington State from late 2021 through 2022, updated Monday and Friday of every week. Our analysis shows that the scraped data is accurate in that it aligns with state administrative hiring data from that period. It also is timelier and offers new insights into teacher demand. Analysis of these data reveals that online postings for special education and English language learner teachers stay open weeks and months longer than those for elementary school teachers, two-thirds of which are filled within two weeks. Schools that serve the largest shares of Black, Hispanic, Native American, Native Hawaiian or Pacific Islander, and multiracial students—a combination we refer to as “underrepresented minority students”—have far greater hiring needs than schools with the smallest shares, and specialized openings remain unfilled for longer periods of time.