News.net is a global news network of more than 110,000 localised sites, from suburb level up to continent, all running on a single heavily customised WordPress platform. We built the ingestion, classification, syndication, media pipeline and elastic infrastructure that lets one central hub feed and coordinate every site.
Nobody documents WordPress at 110,000 sites
WordPress is usually one site. Sometimes a handful. News.net runs more than 110,000 interlinked sites on it, from suburb pages up to continent editions, and at that count the ordinary parts of WordPress start to bite: postmeta and options tables that grow without a ceiling, traffic that spikes with the news cycle, a security surface as wide as the network, and a media library counted in millions of files.
Why WordPress, not a custom platform
News.net started as a prototype while we built towards a custom publishing platform meant to carry the whole network. WordPress was the stand-in. Then we kept adding sites to find the ceiling, and it kept coping, so the custom platform stopped being necessary. People hear WordPress and picture a blog. Underneath it is a competent PHP application: authentication, a REST API, a plugin and theme system, database abstractions, caching hooks. Everything that made News.net hard sat in the infrastructure and the data model, and swapping the CMS out would not have touched either.
Ingestion, classification and a million-item media library
A custom, highly available ingestion platform pulls content from multiple providers, Associated Press and Reuters among them. Each item passes through classification that files it into locality taxonomies, continent down to specific area, in seconds. Media is parsed and transformed into more than 20 sizes and formats to suit every site, device and provider, and a custom API layer syndicates the result. Curators watch content arrive and push it across the network from one system rather than 110,000.
Solving scale on WordPress
The main site acts as a central hub, and every other site can work alone or in concert with it. Sites sit in a hierarchy several levels deep: a Kearneys Spring suburb site has Toowoomba as a parent, then a region, a state and a country above that. Modelling and querying that hierarchy at this size was an engineering problem in its own right. It runs on AWS because the network needs room to move: multi-region for redundancy, separate staging and production environments, load balancers and proxies in front, caching at several layers, and a database tier of multiple read and write clusters rather than one large box. With the database tuning and hardening on top, it holds through ingestion and traffic spikes alike.
Build notes
- More than 110,000 localised sites run from one WordPress platform, with each site able to operate on its own and in unison with the central hub.
- Custom ingestion from providers including Associated Press and Reuters, classified into locality taxonomies from continent down to suburb in seconds.
- A media library of over a million high-resolution items, growing by hundreds daily, with each item transformed into 20+ sizes and formats.
- Elastic AWS infrastructure that reacts within moments to traffic and ingestion spikes.
- A hierarchical model where a suburb site rolls up through region, state and country parents.
- A custom curation and publishing system for distributed teams, plus an API layer that syndicates content to different devices.
