Save a hard-to-open web page as a Google Doc

Give it the address of a page that will not hand its text to anything but a browser, and run it whenever you need one read. A plain request is tried first, and only if that comes back blocked, too short or as a challenge screen is the address read through a reader service that renders it properly. Either way the text lands in a new Google Doc.

You get a document titled after the page, opening with the source address and which route the text came by, followed by the readable text. Each run creates its own document, so nothing you already have is overwritten. The address recorded is rebuilt from the scheme, host and path only: the query string, where an access key travels, is left out, and a failed request is reported by host and reason alone. If neither route produces any text, the run stops rather than saving an empty document.

Use it on pages you are entitled to read: it is for getting at an article you could open yourself in a browser, not for reaching anything you could not.

Set pageUrl to blank.Set pageUrl to blank.

Set minimumCharacters to 600.Set minimumCharacters to 600.

Set maximumCharacters to 20000.Set maximumCharacters to 20000.

Trim pageUrl split by ://[1] and save it as afterScheme.Trim pageUrl split by ://[1] and save it as afterScheme.

Split afterScheme by / and save it as hostAndPath.Split afterScheme by / and save it as hostAndPath.

Trim the first item of hostAndPath and save it as authorityPart.Trim the first item of hostAndPath and save it as authorityPart.

Trim the last item of authorityPart split by @ and save it as pageHost.Trim the last item of authorityPart split by @ and save it as pageHost.

If pageHost is empty or pageHost is nullIf pageHost is empty or pageHost is null

Yes

Stop and report a problem: There is no domain name in the address in the 'pageUrl' step, so there is nothing to fetch.Stop and report a problem: There is no domain name in the address in the 'pageUrl' step, so there is nothing to fetch.

No

No steps

Join the items from index 1 onward of hostAndPath with / and save it as pagePath.Join the items from index 1 onward of hostAndPath with / and save it as pagePath.

Get the first item of pagePath split by ? and save it as pagePath.Get the first item of pagePath split by ? and save it as pagePath.

Get the first item of pagePath split by # and save it as pagePath.Get the first item of pagePath split by # and save it as pagePath.

Set pageScheme to http://.Set pageScheme to http://.

If pageUrl starts with https://If pageUrl starts with https://

Yes

Change pageScheme to https://.Change pageScheme to https://.

No

No steps

Set sourceLabel to pageScheme plus pageHost.Set sourceLabel to pageScheme plus pageHost.

If pagePath isn't emptyIf pagePath isn't empty

Yes

Change sourceLabel to sourceLabel/pagePath.Change sourceLabel to sourceLabel/pagePath.

No

No steps

Set directBody to blank.Set directBody to blank.

Set directOk to false.Set directOk to false.

Set directNote to blank.Set directNote to blank.

If a step below fails, save the error as fetchErrorIf a step below fails, save the error as fetchError

Try

Fetch pageUrl and save it as response.Fetch pageUrl and save it as response.

Trim response.body and save it as directBody.Trim response.body and save it as directBody.

Change directOk to response.ok is true.Change directOk to response.ok is true.

Change directNote to A plain request to pageHost answered with status response.status.Change directNote to A plain request to pageHost answered with status response.status.

On error

Trim fetchError.message and save it as rawReason.Trim fetchError.message and save it as rawReason.

Change directNote to A plain request to pageHost failed: rawReason with pageUrl replaced by the address you gave.Change directNote to A plain request to pageHost failed: rawReason with pageUrl replaced by the address you gave.

Change directOk to false.Change directOk to false.

If directNote contains ://If directNote contains ://

Yes

Change directNote to A plain request to pageHost could not be completed.Change directNote to A plain request to pageHost could not be completed.

No

No steps

Set minimumLength to minimumCharacters multiplied by 1.Set minimumLength to minimumCharacters multiplied by 1.

Set looksBlocked to false.Set looksBlocked to false.

If directOk is noIf directOk is no

Yes

Change looksBlocked to true.Change looksBlocked to true.

No

No steps

If the length of directBody is less than minimumLengthIf the length of directBody is less than minimumLength

Yes

Change looksBlocked to true.Change looksBlocked to true.

No

No steps

If directBody contains just a moment (ignoring case)If directBody contains just a moment (ignoring case)

Yes

Change looksBlocked to true.Change looksBlocked to true.

No

No steps

If directBody contains enable javascript (ignoring case)If directBody contains enable javascript (ignoring case)

Yes

Change looksBlocked to true.Change looksBlocked to true.

No

No steps

If directBody contains captcha (ignoring case)If directBody contains captcha (ignoring case)

Yes

Change looksBlocked to true.Change looksBlocked to true.

No

No steps

If directBody contains access denied (ignoring case)If directBody contains access denied (ignoring case)

Yes

Change looksBlocked to true.Change looksBlocked to true.

No

No steps

If directBody contains attention required (ignoring case)If directBody contains attention required (ignoring case)

Yes

Change looksBlocked to true.Change looksBlocked to true.

No

No steps

If directBody contains verify you are human (ignoring case)If directBody contains verify you are human (ignoring case)

Yes

Change looksBlocked to true.Change looksBlocked to true.

No

No steps

If directBody contains are you a robot (ignoring case)If directBody contains are you a robot (ignoring case)

Yes

Change looksBlocked to true.Change looksBlocked to true.

No

No steps

Set readableText to blank.Set readableText to blank.

Set pageTitle to blank.Set pageTitle to blank.

Set readVia to blank.Set readVia to blank.

If looksBlocked is yesIf looksBlocked is yes

Yes

If a step below fails, save the error as readerErrorIf a step below fails, save the error as readerError

Try

Jina AIRead the page at pageUrl with Jina. Save it as reader.Read the page at pageUrl with Jina. Save it as reader.

Trim reader.content and save it as readableText.Trim reader.content and save it as readableText.

Trim reader.title and save it as pageTitle.Trim reader.title and save it as pageTitle.

On error

Stop and report a problem: Neither a plain request nor the reader service could get the text of that page on pageHost. Check the address, and check you are able to open the page yourself.Stop and report a problem: Neither a plain request nor the reader service could get the text of that page on pageHost. Check the address, and check you are able to open the page yourself.

Change readVia to Read through the reader service, because the plain request did not come back with the page..Change readVia to Read through the reader service, because the plain request did not come back with the page..

No

Change readableText to directBody.Change readableText to directBody.

Change readVia to Read with a plain request. The reader service was not needed..Change readVia to Read with a plain request. The reader service was not needed..

If readableText is nullIf readableText is null

Yes

Change readableText to blank.Change readableText to blank.

No

No steps

If readableText is emptyIf readableText is empty

Yes

Stop and report a problem: That page on pageHost came back with no readable text at all, so there was nothing to save.Stop and report a problem: That page on pageHost came back with no readable text at all, so there was nothing to save.

No

No steps

Set documentTitle to pageHost.Set documentTitle to pageHost.

If pageTitle isn't empty and pageTitle is not nullIf pageTitle isn't empty and pageTitle is not null

Yes

Change documentTitle to pageTitle.Change documentTitle to pageTitle.

No

No steps

Shorten documentTitle to 90 characters and save it as documentTitle.Shorten documentTitle to 90 characters and save it as documentTitle.

Set document to nothing.Set document to nothing.

Set documentId to blank.Set documentId to blank.

If a step below fails, save the error as documentErrorIf a step below fails, save the error as documentError

Try

Google DocsCreate the document documentTitle. Save it as document.Create the document documentTitle. Save it as document.

Trim document.document_id and save it as documentId.Trim document.document_id and save it as documentId.

On error

Stop and report a problem: The page was read, but the Google Doc to put it in could not be created. Check the Google Docs connection and run it again.Stop and report a problem: The page was read, but the Google Doc to put it in could not be created. Check the Google Docs connection and run it again.

If documentId is empty or documentId is nullIf documentId is empty or documentId is null

Yes

Stop and report a problem: Google Docs did not return a document id, so there was nowhere to write the page.Stop and report a problem: Google Docs did not return a document id, so there was nowhere to write the page.

No

No steps

Set header to Source: sourceLabelSet header to Source: sourceLabel readVia .

Change header to header@directNoteChange header to header@directNote .

Set documentText to header plus readableText truncated to a record of kind, op, left, and right characters.Set documentText to header plus readableText truncated to a record of kind, op, left, and right characters.

Google DocsAdd text to documentId.Add text to documentId.documentText

Show in the output: Saved the readable text of pageHost to a new Google Doc.Show in the output: Saved the readable text of pageHost to a new Google Doc.