Starting is easy. Maintaining is not.
Building a simple API connection is straightforward today. Anyone can vibecode a basic connection. But should you? On a daily basis, we talk to customers who built their own pipeline and now are in trouble.So the initial feeling of joy once you vibe code a connection is great. It is very easy to create almost anything today with Claude Code, Codex and similar tools, but at the same time, it has become much harder to maintain all of it. The real cost emerges once pipelines become part of production processes.
So once you vibe code a connection, here's what happens next:
- Reliability — SaaS APIs change without warning, rate limits shift, and auth tokens expire. You'll need retry mechanisms, pagination handling, and multi-format support — none of which came with the original build.
- Observability — Monitoring becomes essential. Regulatory requirements and audits introduce new demands — you need audit trails, data lineage, and proof of what moved where and when. Sensitive data requires identification, masking, access logging and controls. None of this is free to add after the fact.
- Managing changes — APIs change. Schemas drift. What worked last month quietly breaks today. Keeping up means constant maintenance cycles that weren't considered and "priced-in" when the pipeline was first built.
- Access & governance — Who manages this beyond the original developer? Adding multi-user access and permissions is a non-trivial problem — and when that developer moves on, institutional knowledge walks out with them.
- Scale — The initial build handles a certain data volume. Growth means solving for larger throughput, sync/async jobs, timeouts, and parallelism — none of which were in scope originally.
- SLAs & accountability — When a pipeline fails at 2am, who is responsible? Do you have defined recovery times? Can you make commitments to downstream teams or customers about data freshness and availability?
- Extensibility — What happens when you need to onboard a new data source, or acquire a company and need to integrate their entire stack quickly? A bespoke in-house build is rarely designed for that — each addition becomes its own project.
The result: You spend significant time and money, and end up rebuilding something you could have had from day one.
The Organizational Blame Game & Operational Gaps
While the technical debt of custom pipelines is clear, the human and organizational cost is often far more disruptive. When data pipelines are treated as temporary project tasks rather than permanent infrastructure, a predictable series of failures begins:
- The Ownership Vacuum: Most custom pipelines suffer from an ownership gap between "we built it" (the delivery phase) and "we operate it forever." Because no team is explicitly staffed, budgeted, or incentivized to own long-term integration uptime, maintenance falls to the cracks.
- Departmental Finger-Pointing: When a report degradess silently or drops entirely, ownership collapse quickly turns political. In the absence of structured monitoring, IT blames the data team, the data team blames SaaS platform administrators, SaaS platform admins blame IT, and the business consumer loses trust in the dashboards altogether.
- The Partner Profit Drain: For systems integrators and external technology partners, building custom pipelines is a margin killer. What is scoped as a clean handover often turns into a continuous, un-billable escalation loop. The partner finds themselves acting as a 24/7 support desk for upstream SaaS API issues they have no control over, eroding project profitability and strain customer relationships.
- Legacy Blind Spots: Organisations migrating complex legacy databases (such as Sybase, HANA, or DB2) often rely on custom scripts written years ago by engineers who have since departed. Recreating these integrations under a time-sensitive migration schedule forces teams into costly reverse-engineering tasks, bringing unnecessary risk to modern cloud deployments.
The cost argument
Every hour your engineers spend building and maintaining pipelines is an hour not spent on your product. That opportunity cost is rarely visible in a budget — but it compounds.
On top of that, you're paying for your own infrastructure and rebuilding expertise that already exists elsewhere. A managed solution spreads that investment across hundreds of customers — you get the benefit without the overhead. Your team focuses on work that actually differentiates the business.
Side-note on Spark-based pipelines
A common pattern: a data engineer solves a one-off transformation problem using Spark — the tool they know best — and that pipeline gradually gets repurposed for ongoing data movement. This is where it breaks down.
Spark is a transformation engine, not an integration tool. It has no native SaaS connectors and no built-in handling for API authentication, pagination, rate limits, or schema drift. Every source is a custom build. Every API change is a manual fix.
The result is significant operational overhead for something that was never designed for the job — and an infrastructure footprint far larger than the problem warrants.
Category: Industry Insights
