Multiple API Enrichments


#1

I’m attempting to widen some unstructured events coming into Snowplow through the Stream Enrich process. I have two enrichments that are using the API Request Enrichment and they each work when the other is disabled. My other enrichments are working correctly (ip_lookups and ua_parser_config). I’m sure I’m not setting up something correctly. When I’m monitoring the debug logs of the stream enrich I can see the events get processed, but there are no errors recorded.

Here is one of the API enrichments. The other is just like it except using a different input and API uri.

 {
   "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/api_request_enrichment_config/jsonschema/1-0-0",
   "data": {
 	"name": "api_request_enrichment_config",
 	"vendor": "com.snowplowanalytics.snowplow.enrichments",
 	"enabled": true,
 	"parameters": {
 	  "inputs": [
 		{
 		  "key": "batchId",
 		  "json": {
 			"field": "unstruct_event",
 			"schemaCriterion": "iglu:com.acme/some_unstructured_event/jsonschema/1-0-0",
 			"jsonPath": "$.batchId"
 		  }
 		}
 	  ],
 	  "api": {
 		"http": {
 		  "method": "GET",
 		  "uri": "http://api.acme.com/api/v1/snowplow/lookup/batch/{{batchId}}",
 		  "timeout": 5000,
 		  "authentication": {
 			"httpBasic": {
 			  "username": "username",
 			  "password": "password"
 			}
 		  }
 		}
 	  },
 	  "outputs": [ {
 		"schema": "iglu:com.acme/batch/jsonschema/1-0-0",
 		"json": {
 		  "jsonPath": "$.batch"
 		}
 	  } ],
 	  "cache": {
 		"size": 3000,
 		"ttl": 1200
 	  }
 	}
   }
 }

#2

Hi @mialberhasky - you’re not doing anything wrong - unfortunately at the moment the Snowplow pipeline (batch or RT) is limited to running one of each enrichment type.


#3

Is adding support for multiple instances of the same enrichment type on the roadmap? I’m building out our Snowplow pipeline to be used by various teams at my institution and I’m certain that they will eventually want to enhance their collected data with other data sources (api, sql, etc).


#4

Hi Michael - it’s not currently scheduled but there’s a ticket to track this here:

https://github.com/snowplow/snowplow/issues/2260