Hi @alex, thanks for the info. I was meaning to try all of those standard GCP parts as that is usually the canonical setup that google always demo’s for any event streaming example projects. Now I’ll have a good reason to try this out. I guess it does make sense to use BT in some cases especially if you need high throughput for QPS, which BigTable provides, though for dwh type analysis BigQuery has a lot more advantages being a columnar store, having nested data structs and that you don’t need to manage the nodes yourself. Also it supports streaming inserts as well which are not available in Redshift (though it has some caveats too).
The Dataflow seems to be interesting especially if you combine with Apache Beam abstraction to manage the pipelines it might offer best of both worlds not locked cloud option and ability to switch to any other solution.
Will be very interesting to see the RFC for GCP!