Read a whole site into spreadsheet rows

Give it one web address and it works out what else is on that site, reads each page it finds, and adds a row per page to your Google Sheet - so you can summarise a competitor's whole site, or your own, without listing every page by hand.

It finds the pages in one pass. First it asks the site for its sitemap (yours, if you name one in the 'sitemapUrl' step, otherwise '/sitemap.xml'); if there is no usable sitemap - or it is only an index pointing at other sitemaps - it reads the front page once and takes the links on it instead. That is the whole of the discovery: pages found this way are read, but the links on them are never followed, so nothing wanders.

Three limits hold it in place. Every address has to sit under the site you named, matched on the full 'https://host/' prefix, so a link to another site - or to a look-alike domain that merely starts the same way - is dropped rather than fetched. One run reads at most 50 pages however high you set 'maxPages', and the cap is applied to the list before the first page is fetched. And any address already sitting in the sheet's first column is skipped, so running it again picks up where the last run stopped rather than reading everything twice.

Columns are fixed: the page address, then the page title, then your fields in the order you list them in 'fieldsToExtract'. The workflow cannot name your columns for you, so put your own header row in the sheet first - 'Page address', 'Page title', then your fields - and every run lines up underneath it. Values are written exactly as extracted, so nothing off a page can turn into a spreadsheet formula, and rows are only ever added.

A page that will not load, or that comes back with no readable text, is named in the run log and the rest carry on. Fill in 'siteUrl', 'fieldsToExtract' and 'spreadsheetUrl' to set it up, then run it whenever you want the site swept again.