End of life for the MaxMind legacy IP lookups databases [IMPORTANT]

Who this affects

Any Snowplow user enriching their events with the MaxMind-powered IP Lookups Enrichment:

Situation

The original MaxMind GeoIP DAT (legacy) databases are no longer available for new customers, so it’s impossible for a new Snowplow user to get started with those.

Furthermore, MaxMind have announced that:

We will be discontinuing updates to the GeoLite Legacy databases as of April 1, 2018. You will still be able to download the April 2018 release until January 2, 2019. GeoLite Legacy users will need to update their integrations in order to switch to the free GeoLite2 or commercial GeoIP databases by April 2018.

For more information see:

Impact on Snowplow users

The Snowplow IP Lookups Enrichment uses the MaxMind “GeoLite Legacy” databases.

Because the last GeoLite Legacy files will be released on 1st April, effective as of 1st May, your Snowplow installation will be missing out on the latest MaxMind database updates.

Snowplow will continue to function correctly, but the IP Lookups Enrichment will become less and less accurate for incoming events, as IP addresses “drift away” from the 1st April GeoLite Legacy database values.

What we are doing to resolve this

We need to move Snowplow to use the newer format GeoLite2 databases. We have a multiple step plan to do this:

Step 1. Start providing hosting for the GeoLite2 databases

We will start providing this new database file in Amazon S3, as we do the legacy database.

Step 2. Release a new version of scala-maxmind-iplookups

We are working on a new version of this library, with support for the new database file format.

Huge thanks to Snowplow community member Tiago Macedo for providing a PR in support of this work:

Step 3: We will release new Snowplow versions (batch and realtime)

We will update Stream Enrich and Spark Enrich to be able to use the new database file format. You can follow our progress in this milestone:

Preparing your Snowplow installation

As time is relatively tight on this, we strongly recommend open-source users of Snowplow to prepare for this release by upgrading to the most recent available Snowplow release for their respective pipeline (batch or real-time).

Please note that we have no plans to back-port this MaxMind upgrade to any prior versions of Snowplow; if this enrichment is important to your usage of Snowplow, then do upgrade as the first step.

We are globally pinning this thread within Discourse to maximize visibility of this issue, and will post back with updates as our upgrade plan rolls out.

Please let us know any questions.

4 Likes

Hello,

I upgraded my Pipeline to the last versions of both scala stream collector and stream enrich. I checked that the new version of scala-maxmind-iplookups is GeoLite2 database.

I am using Kafka sink for both collector and enrichment to ingest data within topics. When i run my Pipeline, data are stored correctly in the Kafka collector topic but not in enriched one!!

Logs shows this error in the enrichment job:

Exception in thread "main" java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.ObjectNode.<init>(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/Map;)V

Sounds like a jackson-databind version issue?

Can someone please help !

Thanks in advance :slight_smile: