Back to Feed
article 3m read

All-caps text columns kept getting promoted to stores by the importer

A store sheet had short, all-caps columns full of words, and my importer read them as store codes. The change checks a few cells under each candidate, hunts for the real header row, stops putting made-up prices into orders, and handles one cart crash. The new detection rules still have no test of their own.

Article DepthPractice

Flow

What the importer does with each messy input

  1. 1Banner above the header

    Look near the top for a known column name. No match means line one.

  2. 2Short capital heading

    Read a few cells below. A word means it is not a store.

  3. 3Unclear cost

    Take the best source first. Past that it guesses, and doesn't say so.

  4. 4No price at all

    Send zero to the cart, not a made-up number.

  5. 5SKU already taken

    Reuse the existing product. Other errors still show up.

Stores were being made from text columns by the ordering system. One store sheet has a couple of short, all-caps columns full of words, and the importer read their headers as store codes.

Nothing in the file says which column is which, so the importer has to guess. The old guess was based only on how the header looked: short and in capitals meant store, unless the word was on a small ban list. That lets in real stores, but also stray abbreviations, yes/no flags and plain numbers. After that, the cells get read as stock counts. I'd bet a word came out as zero or junk, under a store that isn't real.

Asking the cells instead of the header

The header rule got a little stricter: it needs a letter now, and yes/no words are banned. But the change that counts is a look at the data. For each possible store column, the importer reads a few rows below it. A single word in the data is enough to rule a column out.

A column survives only if every sampled cell is blank or a number.

If that throws out every column on a row, it tries the next row. It's only a few rows, though. A word column with empty top rows still sneaks in, and a real store with one odd note gets dropped without a word.

The detector that runs first

A newer check runs before that one. It trusts a code if the next column is that store's max column. It never reads the cells. If it finds enough of those pairs, it stops there and the older check never runs. They probably agree on the real files. I haven't checked.

The header wasn't on line one either

Some files start with a banner. The parser now looks near the top for a row with a known column name, like a product name or an ID. One match is enough. If none match, it uses line one and doesn't say so.

Prices that looked real

Cost now comes from the best source available, in this order:

  • the vendor's case cost, if there's a vendor row,
  • otherwise the sheet's own case cost,
  • a per-unit cost times case size,
  • a small tag price, treated as per-unit.

Any very low case cost also gets multiplied by case size. The last steps are guesses, and a cheap item could get the wrong price. The saved number doesn't say which rule made it.

Defaults that rode into orders

The step that sent items from the dashboard to the cart gave every product the same made-up case size and price. Those went into orders looking real. Now it sends the imported values, or zero and one when there's nothing. A zero price looks wrong, which is the point.

The cart's error page

Adding an item could crash with a server error, I think because a product already had that SKU under another ID. Now that kind of clash is caught and the existing product is reused. Other errors still show up.

Store names as identity

Stores with the same name now merge into one. That clears out duplicates, but two real stores with one name would collapse together.

Coverage so far

The new tests check the import endpoint and a store create-and-delete flow. Neither one hands the detector a sheet with a word column in it, so the case that started all this still has no test pinning it.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward