Co-founder and CTO of a 150k-transaction/day insurance marketplace shares how building data infrastructure in-house seemed smart until schema changes, broken pipelines, and misallocated engineering talent quietly consumed the business.
When Torchlight, the insurance lead-gen marketplace I co-founded, was growing fast, the last thing on anyone's mind was the data infrastructure. We were closing partners, scaling media spend and expanding our marketplace. The business was working. So we kept building on top of what we had.
The mistake was not stopping to see if there was a better way.
I was co-founder and CTO. I built a lot of the systems myself, and I'll be honest—I thought we had it figured out. AWS-first, everything in-house. PHP scripts for batch transfers, Kinesis and Firehose moving data, Redshift as the warehouse. It looked clean on paper. In reality, it required constant human attention to keep running, and it got harder to manage with every new data source, every schema change, every new buyer requirement we had to accommodate.
At peak, we were handling 150,000 transactions a day, each with 30+ data points. Over 4.5 million individual fields moving through the system every day, all of it needing to be validated and matched in real time. And when the system couldn't keep up, we felt it everywhere.
The pitch for building your own data infrastructure sounds reasonable: you control it, you own the institutional knowledge, no vendor dependencies. What that pitch leaves out is the ongoing cost of maintaining something you built for the business you had, not the business you're becoming.
Every time we added a new field to our transactional database, it rippled through the entire stack. Scripts, pipelines, warehouse tables—all of it needed to be updated manually. That's not a one-time engineering task. That's a recurring tax on your team's time, and it balloons as the system grows more complex.
We also had the wrong people doing it. We leaned on software developers to maintain data pipelines because that's who we had. But software developers and data engineers are not interchangeable. We were paying senior developer rates for work that a data engineer would have handled more effectively and cheaper. That's not a knock on the team; it's a structural problem we should have caught earlier.
The result was predictable in hindsight: Our best engineers were spending their days monitoring dashboards and fixing broken scripts instead of building things. I wasn't immune to it, either. I ended up personally maintaining legacy scripts that I had no business touching at that stage of the company. Every hour spent on that was an hour not spent on what actually mattered.
I don't think Torchlight's situation was unusual. Most fast-growing companies build their data infrastructure the same way we did—reactively, tool by tool, patching gaps as they appear. It works until it doesn't, and by the time it stops working, you're already behind.
The underlying issue is that most data systems are designed around the current state. They're built to handle the volume and schema you have today. They're not built to absorb change cheaply. So every time the business evolves with new products, partners and data sources, the system resists rather than adapts. And the faster the business grows, the more often that happens.
What we needed was infrastructure designed around the assumption that things will change, not the hope that they won't. Pipelines that handle schema evolution without breaking. The ability to replay data when something goes wrong rather than just accepting the gap. Engineering teams that are building, not babysitting.
None of that is exotic. The tooling exists. The practices exist. What's harder is convincing a fast-moving team to invest in it before the pain is obvious—because by the time it's obvious, you're already spending most of your engineering capacity just keeping the lights on.
You probably don't need a full audit to know if your data infrastructure is lagging behind your business. A few things to honestly assess:
The good news is that none of this requires starting over. Most teams are closer to a fix than they think; they just need to stop treating infrastructure problems as background noise.
For more, check out my “Lessons in Data Movement” whitepaper.
Andy Guy is a technology leader, entrepreneur, and investor with more than 15 years of experience building and scaling data-driven businesses. He co-founded Torchlight Technology Group, growing it from a three-person startup into a 40+ employee company and leading its successful acquisition. At Torchlight, Andy served as CTO, architecting the company’s proprietary platforms for digital marketing and insurance lead generation. He has since held technology leadership roles with Intellivets and SunshineMD, and today is EASL’s Head of Product.
We're data geeks who love to chat with anyone who appreciates clean infrastructure and issue-free data streams.
