Implementing two consecutive API enrichments

For engineers
1

Hello everyone,

I am trying to implement a pipeline with this kind of architecture:

Tracker > Kafka collector > Kafka topic no1 > Kafka enricher (with api enrichment) > Kafka topic no2 > Kafka enricher (with another api enrichment) > Kafka topic no3

First of allo is it possible for an enricher to consume an already enriched event and perform another enrichment? I have trouble with deserialization because i think the results of the first enrichment are in a TSV format and the second enricher awaits data in thrift format.

How is it possible to perform two consecutive API enrichments for the same event?

Thank you in advance!

2

Hey @bambachas79,

i think the results of the first enrichment are in a TSV format and the second enricher awaits data in thrift format.

Yup, this is correct. Every step assumes a certain input format and produces certain output format. So, unfortunately you cannot do that. At the moment you also cannot have two independent API Request enrichments in a single enrich step, but this is something we’d like to fix in future. So far the only way to have two independent enrichments is to encode one of them as API Request enrichment and another one in JS enrichment (which basically can do anything) or both as a single JS enrichment with two-step logic.

3

So i need two independent enrichers with independent configuration files that are consuming the same event and perform two different api enrichments, right?

4

That’s one more way, yes. But very cost-inefficient one I’d say, especially compared to JS enrichment.

2 Likes
5

How is it possible to send an http request from a JS enrichment without requiring any packages?
i am getting an error:

Evaluating JavaScript script threw an exception: [org.mozilla.javascript.EcmaError: ReferenceError: \"require\" is not defined. (user-defined-script#1)]

I am trying to use the http library like that:

const https = require('https')

I also tried to write down inside the script a whole minified library to make an http request and i got some other errors.

6

JavaScript enrichment uses Rhino JS engine, which is basically “JavaScript inside JVM”, which means you should have access to the whole Java 8 standard library and libraries bundled with Scala Common Enrich, but you cannot use anything modules-like in your JS snippets.

For example, here’s a snippett for HTTP request using Apache Commons: https://stackoverflow.com/questions/22157910/using-rhino-js-engine-to-make-http-requests/30487571

I have a vague memory of somebody posting a similar question and solution on this forum, but couldn’t find it.

1 Like
7

I just figured it out after posting the question, i am sorry.
I am already checking it! Thank you very much! :slight_smile: