Callrails webhook breaking during enrich

For engineers Enrichment
1

I tried to get the callrails webhook to work, but only see

{
  "schema": "iglu:com.snowplowanalytics.snowplow.badrows/adapter_failures/jsonschema/1-0-0",
  "data": {
    "processor": {
      "artifact": "beam-enrich",
      "version": "1.2.0"
    },
    "failure": {
      "timestamp": "2020-07-30T01:03:04.160Z",
      "vendor": "com.callrail",
      "version": "v1",
      "messages": [
        {
          "field": "querystring",
          "value": null,
          "expectation": "empty querystring"
        }
      ]
    },
    ...

in the bad enrichment pipeline.

I am curling it with a sample webhook that I took from call rails

{
  "answered": true,
  "business_phone_number": "+16029183127",
  "call_type": "answered",
  "company_id": 996192227,
  "company_name": "1234567 - Digital Pharmacist Test",
  "company_time_zone": "America/Chicago",
  "created_at": "2020-03-09T15:03:58.705-04:00",
  "customer_city": "Natchitoches",
  "customer_country": "US",
  "customer_name": "Terrell Robert",
  "customer_phone_number": "+13186520010",
  "customer_state": "LA",
  "device_type": "",
  "direction": "inbound",
  "duration": "550",
  "first_call": true,
  "formatted_call_type": "Answered Call",
  "formatted_customer_location": "Natchitoches, LA",
  "formatted_business_phone_number": "602-918-3127",
  "formatted_customer_name": "Terrell Robert",
  "prior_calls": 0,
  "formatted_customer_name_or_phone_number": "Terrell Robert",
  "formatted_customer_phone_number": "318-652-0010",
  "formatted_duration": "9m 10s",
  "formatted_tracking_phone_number": "512-539-0164",
  "formatted_tracking_source": "Digital Pharmacist",
  "formatted_value": "--",
  "good_lead_call_id": "",
  "good_lead_call_time": "",
  "id": 675725529,
  "lead_status": "",
  "note": "",
  "recording": "",
  "recording_duration": "",
  "source": "Digital Pharmacist",
  "source_name": "Digital Pharmacist",
  "start_time": "2020-03-09T15:03:58.477-04:00",
  "tags": [],
  "total_calls": 1,
  "tracking_phone_number": "+15125390164",
  "transcription": "",
  "value": "",
  "voicemail": false,
  "waveforms": "",
  "tracker_id": 586544935,
  "keywords": "",
  "medium": "",
  "referring_url": "",
  "landing_page_url": "",
  "last_requested_url": "",
  "referrer_domain": "",
  "conversational_transcript": "",
  "utm_source": "",
  "utm_medium": "",
  "utm_term": "",
  "utm_content": "",
  "utm_campaign": "",
  "utma": "",
  "utmb": "",
  "utmc": "",
  "utmv": "",
  "utmz": "",
  "ga": "",
  "fbclid": "",
  "gclid": "",
  "integration_data": [
    {
      "integration": "Webhooks",
      "data": null
    }
  ],
  "keywords_spotted": "",
  "recording_player": "",
  "speaker_percent": "",
  "call_highlights": [],
  "agent_email": "",
  "campaign": "Website",
  "msclkid": "",
  "callercity": "Natchitoches",
  "callercountry": "US",
  "callername": "Terrell Robert",
  "callernum": "+13186520010",
  "callerstate": "LA",
  "callsource": "all",
  "custom": "",
  "datetime": "2020-03-09 19:03:58",
  "destinationnum": "+16029183127",
  "kissmetrics_id": "",
  "landingpage": "",
  "referrer": "",
  "referrermedium": "",
  "score": "",
  "tag": "",
  "trackingnum": "+15125390164",
  "timestamp": "2020-03-09T15:03:58.477-04:00",
  "person_id": 602583854,
  "person_resource_id": "PERcf22544388634589864989ee0b090a50",
  "resource_id": "CAL9b3e8bebc434446c9a1fbdee6e7d7de2",
  "company_resource_id": "COM4364e5f9426a4da3b262420c3563e56b",
  "tracker_resource_id": "TRKaa503df80c5246fc8f85d0f6e4f1710b"
}

using

curl localhost:9001/com.callrail/v1?aid=test -H 'content-type: application/json' -d @./examples/callrails/test.json

I am not sure what that error means or what I am doing wrong. I tried to follow https://docs.snowplowanalytics.com/docs/collecting-data/collecting-data-from-third-parties/callrail/

2

Hi @camerondavison

I tried to recreate your error - I ran snowplow micro locally and then I ran your curl command with your attached json.

I also got a bad row, but it’s different to the one you reported:

{
    "schema": "iglu:com.snowplowanalytics.snowplow.badrows/schema_violations/jsonschema/2-0-0",
    "data": {
        "processor": {
            "artifact": "snowplow-micro",
            "version": "0.1.0"
        },
        "failure": {
            "timestamp": "2020-07-30T16:02:30.625Z",
            "messages": [
                {
                    "schemaKey": "iglu:com.callrail/call_complete/jsonschema/1-0-2",
                    "error": {
                        "error": "ValidationError",
                        "dataReports": [
                            {
                                "message": "$.datetime: is missing but it is required",
                                "path": "$",
                                "keyword": "required",
                                "targets": [
                                    "datetime"
                                ]
                            },
                            {
                                "message": "$.id: is missing but it is required",
                                "path": "$",
                                "keyword": "required",
                                "targets": [
                                    "id"
                                ]
                            }
                        ]
                    }
                }
            ]
        },
        "payload": {}
    }
}

(I curtailed payload because it is very long)

This bad row makes sense to me, because I can see from the schema definition that id should be a string and datetime is required.

But I cannot explain why you got an adapter failure with your request, but I got a schema violation with the exact same request. It might just be a problem with your local setup, and whatever you have running on your local port 9001. From your error message it looks like your query string ?aid=test is not being propagated to the collector.

3

Hmm. I see datetime in there. Am I looking at that correctly.
I pulled this straight from a real request from callrails, Does that mean that the schema needs to change to accept both int and string for the id?

Looking at the response that you got makes me think that its expecting everything to be wrapped in something and that is why it did not find the 2 required fields.

4

@camerondavison, the “datetime” requires adhering to ISO8601 format as per JSON schema specification. The corresponding JSON schema is here.

5

I was able to get your same error after fixing the akka configuration.

Interestingly enough I was able to get

❯ curl -v -XPOST 'https://localhost:9001/com.callrail/v1?aid=test&id=675725529&datetime=2020-03-09T15:03:58.477-04:00'

to work. so is there another akka setting that I am missing to pass along the body and the url params.

6

I ran the micro as well and tried

❯ curl -H 'content-type: application/json; charset=utf-8' -d'{"id":"675725529","datetime":"2020-03-09T15:03:58.477-04:00"}' -v http://localhost:9090/com.callrail/v1

which give me

  {
    "collectorPayload": {
      "api": {
        "vendor": "com.callrail",
        "version": "v1"
      },
      "querystring": [],
      "contentType": "application/json",
      "body": "{\"id\":\"675725529\",\"datetime\":\"2020-03-09T15:03:58.477-04:00\"}",
      "source": {
        "name": "ssc-0.15.0-stdout$",
        "encoding": "UTF-8",
        "hostname": "localhost"
      },
      "context": {
        "timestamp": "2020-08-07T02:56:24.325Z",
        "ipAddress": "172.17.0.1",
        "useragent": "curl/7.64.1",
        "refererUri": null,
        "headers": [
          "Host: localhost:9090",
          "User-Agent: curl/7.64.1",
          "Accept: */*",
          "Timeout-Access: <function1>",
          "application/json"
        ],
        "userId": "aa1053d9-502b-4b1a-a63b-0c0c3ec0eebc"
      }
    },
    "errors": [
      "Error while extracting event(s) from collector payload and validating it/them.",
      "Querystring is empty: no CallRail event to process"
    ]
  }

and

❯ curl -H 'content-type: application/json; charset=utf-8' -d'{"id":"675725529","datetime":"2020-03-09T15:03:58.477-04:00"}' -v http://localhost:9090/com.callrail/v1?aid=test

gave me

{
    "collectorPayload": {
      "api": {
        "vendor": "com.callrail",
        "version": "v1"
      },
      "querystring": [
        "aid=test"
      ],
      "contentType": "application/json",
      "body": "{\"id\":\"675725529\",\"datetime\":\"2020-03-09T15:03:58.477-04:00\"}",
      "source": {
        "name": "ssc-0.15.0-stdout$",
        "encoding": "UTF-8",
        "hostname": "localhost"
      },
      "context": {
        "timestamp": "2020-08-07T02:56:56.610Z",
        "ipAddress": "172.17.0.1",
        "useragent": "curl/7.64.1",
        "refererUri": null,
        "headers": [
          "Host: localhost:9090",
          "User-Agent: curl/7.64.1",
          "Accept: */*",
          "Timeout-Access: <function1>",
          "application/json"
        ],
        "userId": "640fb1db-287b-430a-a878-437e9886bf92"
      }
    },
    "errors": [
      "An error occured while extracting the info about the event.",
      "Error while extracting the schema of the event: object has missing required properties ([\"datetime\",\"id\"])"
    ]
  }

It seems like it does not support the json payload?

7

The tests in the enricher suggest that the data is sent as GET (so I’m not sure if POST is supported). Could you try and adapt a cURL request from this test and see if that works?

8

Ah yes you are right. I think i took that json from an example from their API but that is not what the webhook is sending. It is sending

/com.callrail/v1?answered=false&callercity=Houston&callercountry=US&callername=Davison+Cameron&callernum=%2B15551231234&callerstate=TX&callsource=all&campaign=Website&custom=&customer_phone_number=%2B15551231234&datetime=2020-08-07+14%3A00%3A10&destinationnum=%2B15127319128&device_type=&duration=35&first_call=false&ga=&gclid=&id=359004955&keywords=&kissmetrics_id=&landingpage=&recording=&referrer=&referrermedium=direct&resource_id=CAL304942c7686749f0af0ee0adda063d27&score=&trackingnum=%2B15125390164&utm_campaign=aa&utm_content=&utm_medium=direct&utm_source=website&utm_term=&utma=&utmb=&utmc=&utmv=&utmz='

but then i am seeing the following “bad row”

      "Error while extracting the schema of the event: object instance has properties which are not allowed by the schema: [\"campaign\",\"custom\",\"customer_phone_number\",\"device_type\",\"resource_id\",\"score\"]"

I guess that just means that the callrails webhook schema is out of date?

9

@camerondavison, looks like only GET version works but not POST. For simplicity, adding only the required fields works fine for me

curl --request GET \
  --url 'https://com-snowplowanalytics-dev1.mini.snplow.net/com.callrail/v1?aid=test&id=1234567&datetime=2020-03-09T19%3A03%3A58%2B00%3A00' \
  --header 'content-type: application/json'

If there are new fields added automatically you might need to host your own version of the JSON schemas to be able to track those.