All of these tools are insanely expensive (from my own experience at companies that have used them). I understand it, since building your own pipeline to handle the kind of throughput analytics takes is expensive and time consuming. Business leaders want the visibility but don't want to redirect dev resources to build and maintain these creaky data pipelines. It is the perfect market of high-value and low tolerance for build (on the build or buy spectrum).
But I am not going to pay $1000/month as a bootstrap startup. What open source alternatives exist that can be run on basic hardware?
The best open source options are Airbyte and Meltano / Singer. But it's hard to keep them running. If you self-host them, you'll hit issues at least a few times a month which can each take a few hours to solve.
It's not like running Postgres which "just works". When you self-host Airbyte, you're still building a good bit.
I felt the same way about the cost of data tools. Paying $1,000 for Fivetran, $2,000 for Snowflake, $2,000 for Looker seemed crazy. We bundle all three for $500 / month at https://www.definite.app
Hi, I've been loking something like this! Any of your custumer has success story migrating off bigquery to your platform? And how do you compare to motherduck? (Looks like you built some of ypur stack on top of duckdb)
Your comment reads like a pitch but I checked definite and I've been looking for something like this. What we you using it for? Did you evaluate any other Ai analytics tools?
I'm not sure about Census but Fivetran's free plan has met my needs to sync data from different ad platforms to BigQuery pretty well.
One of their pitfalls is charging by the row. If you're cost-conscious, you really need to watch what data you're syncing and you need to pare it down quite a bit during the 2-week period they give you when setting up a new connector. If you do all that though, you can get a lot of mileage out of the free plan for some use cases.
Or batch massive rows? JSON structures in-database go a long way...
Check out redpanda connect / warpstream bento (depending on your license needs). Both came out of what was benthos.
Interesting, it looks like redpanda is a Kafka replacement and redpanda connect is a Kafka connect replacement but with a supported set of connectors (sources and sinks). I (once upon a time) had to write a Kafka connector myself so I get the general idea.
To be honest, I hadn't really given much thought about what event streaming I would use anyway. So I imagine using redpanda along with redpanda connect could be that layer (I was considering just using Redis streams or even PostgreSQL) and then there is just another redpanda connector for the db to add into that mix. If someone is starting from scratch that might be a good path. But I agree the MIT license of warpstream is a bit nicer if all you need is the connectors.
I've tried airbyte, sling, and dlt (besides building several tools from scratch)
My best bet for now will be dlt if you have dedicated DE team, but sling will get you a long way for moving data around your warehouse
Haven't used it personally, but I would suggest looking into Apache Hudi
Good to know about but looks more like an open source snowflake (e.g. data lake). Fivetran and Census are the in/out process layers that bookend the data lake. Although, Hudi does look like it has some of that functionality baked in.
Airbyte is probably the best opensource tool in this space.
Airbyte gave us more headaches than it was worth. But if you can get it to work for you, it's probably the closest you'll get to Fivetran in the open-source tool space
Cheers, that is what I was thinking must exist but didn't know about.
Ok if you're bootstrap it probably doesn't make sense but otherwise fivetran is fantastic for not having to deal with a boatload of third parties constant API updates and changes. If your core competency is something else entirely and not doing ETL, then it's worth paying for so you're not wasting time on doing that ETL work.
Yes, I've used Fivetran at VC funded startups that I worked at and I understand the value of not having to build this piece of common infrastructure. Although we did experience regular (probably once every couple of months) issues with our ETL getting out of sync. We even had to do a full re-sync on a couple of occasions (which to their credit they did for no charge).
As I said, I totally understand this market and why these companies are valuable. I respect the work they do. But while I am a tiny, tiny startup I don't want to lock in to anything and I know I can handle the amount of data myself with little effort if I have a basic open source alternative I can manage myself.
A bootstrapped startup needs a MySQL database and a bunch of SELECT queries. Everything else is overkill.
Sure, SQL + something like metabase is a decent starting point (ideally running on a read-only replica). However, there is room to improve over that.
It's like logging. Yeah, there is sentry, papertrail, splunk, datadog and the like. But something better than greping sys logs is nice and totally reasonable for a startup to standup with Kibana/Elastic running on a tiny instance. That can provide significantly higher value.
There is a middle ground between stone tools and jet aircrafts. I was asking: what are the middle ground tools in this space.
Palantir's market cap is $274b and they make glorified dashboards. There's just too much money in it to spend cycles doing it for free.
This indeed sounds like closing the loop, congrats to the team! Boris’s announcement is pretty interesting: https://www.getcensus.com/blog/census-joins-forces-with-five...
Congrats to everyone. Some of the smartest and kindest people in data coming together!
Fivetran has been great. But in this new ai world. Something like dragster + dlt and sling. You can have your own fivetran developed in house. I haven’t dove too much into reverse etl- but it would be awesome to see a dtl like open source tool for reverse etl.
Congrats to both Census and Fivetran. Census has an amazing product and very good people. Excited to see what's coming next from y'all
Fivetran should’ve done this a long time ago. I think that both etl and reverse etl is going open source route. With this ai world we live in now. You just need dagster or temporal - and a few lines of python.
Seems like a no-brainer. I wonder if they ever started to build these capabilities in house; I'm sure they already had so much of the tooling available.
Yeah, I was always curious why Fivetran didn't build this themselves when reverse-ETL started to take off.
I built a company[0], SeekWell, in this space (launched before Census), but was mostly focused on Sheets and Slack as destinations. SeekWell was acquired a few years ago too.
The challenge of syncing from stubborn SaaS tools to your data warehouse / database I suspect is different than syncing data from your data warehouse / database back to SaaS tools. Specifically, reverse ETL has to incorporate more context from the business I guess so the data that lands in the 3rd party tools is actually solid.
Once you have customers and a good network of integrations with a large number of tools, I suspect it's easier to just buy that company than build it all yourself?
This is exactly right. We even went so far as to build a proof of concept internally, and the technical challenges are just very different. The simplest way to explain it is that Fivetran connects a skinny pipe (APIs) to a fat pipe (databases) while Census connects a fat pipe to a skinny pipe.
The data is only as solid as you make it to be. Ultimately reverse ETL is just a technology (basically from SQL to APIs). The quality/correctness of data is someone else's headache. I've been there and done that, and reverse ETL is a feature-product with huge churn. See how Hightouch pivoted hard from that into CDP.
Is Fivetran to Fortran like what C++ is to C?
It’s like what Cantonese is to cantaloupe
Anyone hear rumors of how much they were acquired for?
I'd guess a little north of $500 million.
- Census last raised $60M Series B at a $630M valuation (upper bound)
- Census’s estimated annual revenue is $31.6 million with ~200 employees.
- Median private-SaaS EV/ARR multiple is 7× (7 * 31 = 217 = lower bound)
- Hightouch raises $80M on a $1.2B valuation(at ~60× ARR)
- Twilio completes $3.2B acquisition of Segment at ~21× ARR (upper multiple bound)
What does this actually mean for customers? Is are we going to have to rebuild our Census syncs in Fivetran or will the product continue to run as-is? Will plans / pricing change?
Congrats to the Census team and the Fivetran team!
Thanks DU! You've had our back since the early days, thanks for taking a chance on us!
Always wondered FT and Census might come into the each other's territory. Good to see both are merging forces together now!
Still missing a critical piece - ETL. If they acquire it instead of just asking people to use dbt, they have (kinda) the full stack solution.
Congrats to the teams! Like others have said, your pricing ends up killing adoption for my company. We ended up self-hosting Airbyte. It ain't perfect but at least we're not paying $10/GB to replicate data within our own VPC.
there's going to be more consolidation in data tooling this year. Many of the stand alone tools raised too much money and no one wants to buy 5 really expensive tools to assemble a "data stack" anymore.
if you want a data platform that's built to work as one cohesive unit, we got you: https://www.definite.app/
Definite has a data lake, ETL, and BI in one app.
not a big fan of fivetran's pricing
I run a professional services org that helps you switch to an open source alternative. We'll host the solution for you if you want and aim to be drop-in Fivetran compatible in your workflows with a transition plan so you can run the thing if you'd like. Pricing is flexible. Personal email in profile.
Can you say more? Is it the unpredictability, or just too expensive?
They charge based on the highest number of unique rows transferred for the month. It gets expensive quickly.