From 50MB to 38KB: what real data does to a prototype

The heaviest page on a reinsurance underwriting platform I built was sending 50.4MB to the browser. It took around 4.6 seconds when it worked, and on the biggest submissions it timed out.

After the fix, the same page sent 38.5KB. That’s more than 99.9% smaller.

Test data lies

The platform prices insurance on property portfolios. Each submission lists the locations being covered.

The median live submission had 26 locations. The largest had 13,922. Everything worked on the median. The largest is where it fell over.

What was actually wrong

The 50.4MB was two duplicate responses, of 23.2MB and 27.2MB. The page was fetching much the same data twice, and a lot of it.

The fix removed the duplication and cut the response down to 38.5KB.

It wasn’t the only one

Real submissions found more. I also fixed:

• A worker running out of memory on a spreadsheet that arrived with an 889MB cache inside it.
• An integer overflow on a value in a submission that was bigger than the database column could hold.
• Database deadlocks that surfaced to users as server errors.

None of these were bugs in the usual sense. The code did what it was written to do. The data was just bigger and stranger than anyone had planned for.

What a team gets from this

A prototype and a test suite built on typical data will both tell you everything’s fine. Production data is where the product actually gets tested, and it helps to have someone who goes looking for the worst case before a customer finds it.

If your product is about to meet real data, let’s talk.