Feed cleaner stripped tags before they existed

Flow

The cleaner, step by step

1. 11. Undo the codes⌄

   One layer per pass, ampersand code last, stop when nothing changes or at a small cap. Check: a paragraph and link coded twice become real tags.

2. 22. Space for block tags⌄

   Paragraphs, list items and headings become a space; links and bold vanish with no space. Check: a list after a sentence keeps a space before it.

3. 33. Strip the tags⌄

   Every remaining tag comes out, now that the hidden ones are visible.

4. 44. One last decode⌄

   Turns plain ampersand codes back into the character. Check: an ampersand in the text shows as the character, not the code.

5. 55. Bump the parser version⌄

   Per the reader docs, sources then fetch again in full so old rows get reparsed. Check: an old row shows the new preview after its next fetch.

The cleaner in the new RSS reader was supposed to keep HTML out of the story previews. It did its two jobs in the wrong order.

First it stripped out tags. Then it turned codes like the one for an ampersand back into real characters. But lots of feeds send their HTML in code form so it can sit inside XML. Some send it in code form twice.

Why the stripper found nothing to strip

The XML reader undoes one layer of codes. Anything deeper reaches the cleaner still in code form.

So the stripper looked for tags and found none, just ampersands and letters. It let everything through. Then the decode step turned the codes into real tags. The step that removes tags ran before the tags existed.

The old decoder also handled the ampersand code first. My reading is that this let one pass undo two layers at once, but I haven't run the old code to check.

Peeling one layer per pass

Now decoding comes first, in a loop. Each pass undoes one layer, because the ampersand code goes last. The loop stops when a pass changes nothing, or after a small cap.

for _ in range(limit):
    peeled = unescape_one_layer(text)
    if peeled == text:
        break
    text = peeled

I want that cap so nobody can make the parser do a thousand passes. Text coded more times than the cap keeps some junk, and I'm fine with that.

Then the stripper runs and can see the real tags. Then one more decode, because I'm fairly sure the stripper turns plain ampersands back into codes.

The test is a feed item with a paragraph and a link, coded twice. It expects one clean sentence.

The space that tag removal deleted

With the tags gone, a paragraph and a list after it would run together, last word glued to first word. A follow-up change puts a space where block tags were, like paragraphs, list items and headings. Links and bold text still vanish with no space, so a word with one bold letter stays whole. The test gained a list item and expects a space before it.

Titles and links share the helper

The same cleaner handles titles and link addresses too, so a mistake here reaches more than previews.

Rows the parser already wrote

Fixing the parser doesn't fix rows it already saved. Each change bumped the reader's parser version. The reader's docs say a bump makes the sources fetch again in full, with no "nothing changed" shortcut. Without that bump, as far as I understand it, the old broken previews would have stayed broken.