Looking at some of the data it starts to feel more and more like a potential bug in either the operating system or potentially the hardware itself (some of these devices have very low cost RTCs) rather than the trackers.
Yeah I’m starting to feel this way too. While POSTGRES might be the explanation, it strikes me as super weird that it’s only that field that gets affected. Surely if postgres encoding were at play here we’d see it happen for any similarly typed fields…
I guess alternatively would it make sense for the enricher to be able to filter out these events?
From the purist point of view, I’m not sure it does - validation only checks types and specifications depicted by JSONSchema - this data is technically valid, it just has weird values. One might consider that we could have some kind of ‘minimum’ constraint here, but a big part of the philosophy is that data from any time can be reprocessed again - so in theory any such constraint conflicts a bit with design. So while I see the point of view, personally I’d be hesitant to do this in the core validation of the pipeline (if it were my choice at least).
From the practical point of view I think it’s an absolutely fair ask and it’s a good idea - custom validation rules are an idea that has been raised to us before in a different context, and that idea is one that we’re enthusiastic about. At the moment, it’s not on the roadmap, as we have a lot of more immediate delivery - so it’s still a question of if not when. (For the record, I’m not involved in making roadmap decisions, I’m just giving my opinion on how things look wrt that feature). Personally I’m super excited about the idea though!
They are showing up as bad big query inserts since that table is partitioned by time and they are not allowing us to send them to big query.
I would say that this is not intuitive but is actually the expected behaviour. If you’d like to work around it, setting the
true_tstamp will override the
derved_tstamp to be what you set it to (derived_tstamp is normally calculated using dvce_created, dvce_sent and collector tstamps).
Additionally, as I was responding to this I realised that there’s potentially a good solution of the general case of this problem so I’ve created an issue on that topic.