Callrails webhook breaking during enrich

I tried to get the callrails webhook to work, but only see

{
  "schema": "iglu:com.snowplowanalytics.snowplow.badrows/adapter_failures/jsonschema/1-0-0",
  "data": {
    "processor": {
      "artifact": "beam-enrich",
      "version": "1.2.0"
    },
    "failure": {
      "timestamp": "2020-07-30T01:03:04.160Z",
      "vendor": "com.callrail",
      "version": "v1",
      "messages": [
        {
          "field": "querystring",
          "value": null,
          "expectation": "empty querystring"
        }
      ]
    },
    ...

in the bad enrichment pipeline.

I am curling it with a sample webhook that I took from call rails

{
  "answered": true,
  "business_phone_number": "+16029183127",
  "call_type": "answered",
  "company_id": 996192227,
  "company_name": "1234567 - Digital Pharmacist Test",
  "company_time_zone": "America/Chicago",
  "created_at": "2020-03-09T15:03:58.705-04:00",
  "customer_city": "Natchitoches",
  "customer_country": "US",
  "customer_name": "Terrell Robert",
  "customer_phone_number": "+13186520010",
  "customer_state": "LA",
  "device_type": "",
  "direction": "inbound",
  "duration": "550",
  "first_call": true,
  "formatted_call_type": "Answered Call",
  "formatted_customer_location": "Natchitoches, LA",
  "formatted_business_phone_number": "602-918-3127",
  "formatted_customer_name": "Terrell Robert",
  "prior_calls": 0,
  "formatted_customer_name_or_phone_number": "Terrell Robert",
  "formatted_customer_phone_number": "318-652-0010",
  "formatted_duration": "9m 10s",
  "formatted_tracking_phone_number": "512-539-0164",
  "formatted_tracking_source": "Digital Pharmacist",
  "formatted_value": "--",
  "good_lead_call_id": "",
  "good_lead_call_time": "",
  "id": 675725529,
  "lead_status": "",
  "note": "",
  "recording": "",
  "recording_duration": "",
  "source": "Digital Pharmacist",
  "source_name": "Digital Pharmacist",
  "start_time": "2020-03-09T15:03:58.477-04:00",
  "tags": [],
  "total_calls": 1,
  "tracking_phone_number": "+15125390164",
  "transcription": "",
  "value": "",
  "voicemail": false,
  "waveforms": "",
  "tracker_id": 586544935,
  "keywords": "",
  "medium": "",
  "referring_url": "",
  "landing_page_url": "",
  "last_requested_url": "",
  "referrer_domain": "",
  "conversational_transcript": "",
  "utm_source": "",
  "utm_medium": "",
  "utm_term": "",
  "utm_content": "",
  "utm_campaign": "",
  "utma": "",
  "utmb": "",
  "utmc": "",
  "utmv": "",
  "utmz": "",
  "ga": "",
  "fbclid": "",
  "gclid": "",
  "integration_data": [
    {
      "integration": "Webhooks",
      "data": null
    }
  ],
  "keywords_spotted": "",
  "recording_player": "",
  "speaker_percent": "",
  "call_highlights": [],
  "agent_email": "",
  "campaign": "Website",
  "msclkid": "",
  "callercity": "Natchitoches",
  "callercountry": "US",
  "callername": "Terrell Robert",
  "callernum": "+13186520010",
  "callerstate": "LA",
  "callsource": "all",
  "custom": "",
  "datetime": "2020-03-09 19:03:58",
  "destinationnum": "+16029183127",
  "kissmetrics_id": "",
  "landingpage": "",
  "referrer": "",
  "referrermedium": "",
  "score": "",
  "tag": "",
  "trackingnum": "+15125390164",
  "timestamp": "2020-03-09T15:03:58.477-04:00",
  "person_id": 602583854,
  "person_resource_id": "PERcf22544388634589864989ee0b090a50",
  "resource_id": "CAL9b3e8bebc434446c9a1fbdee6e7d7de2",
  "company_resource_id": "COM4364e5f9426a4da3b262420c3563e56b",
  "tracker_resource_id": "TRKaa503df80c5246fc8f85d0f6e4f1710b"
}

using

curl localhost:9001/com.callrail/v1?aid=test -H 'content-type: application/json' -d @./examples/callrails/test.json

I am not sure what that error means or what I am doing wrong. I tried to follow https://docs.snowplowanalytics.com/docs/collecting-data/collecting-data-from-third-parties/callrail/

Hi @camerondavison

I tried to recreate your error - I ran snowplow micro locally and then I ran your curl command with your attached json.

I also got a bad row, but it’s different to the one you reported:

{
    "schema": "iglu:com.snowplowanalytics.snowplow.badrows/schema_violations/jsonschema/2-0-0",
    "data": {
        "processor": {
            "artifact": "snowplow-micro",
            "version": "0.1.0"
        },
        "failure": {
            "timestamp": "2020-07-30T16:02:30.625Z",
            "messages": [
                {
                    "schemaKey": "iglu:com.callrail/call_complete/jsonschema/1-0-2",
                    "error": {
                        "error": "ValidationError",
                        "dataReports": [
                            {
                                "message": "$.datetime: is missing but it is required",
                                "path": "$",
                                "keyword": "required",
                                "targets": [
                                    "datetime"
                                ]
                            },
                            {
                                "message": "$.id: is missing but it is required",
                                "path": "$",
                                "keyword": "required",
                                "targets": [
                                    "id"
                                ]
                            }
                        ]
                    }
                }
            ]
        },
        "payload": {}
    }
}

(I curtailed payload because it is very long)

This bad row makes sense to me, because I can see from the schema definition that id should be a string and datetime is required.

But I cannot explain why you got an adapter failure with your request, but I got a schema violation with the exact same request. It might just be a problem with your local setup, and whatever you have running on your local port 9001. From your error message it looks like your query string ?aid=test is not being propagated to the collector.

Hmm. I see datetime in there. Am I looking at that correctly.
I pulled this straight from a real request from callrails, Does that mean that the schema needs to change to accept both int and string for the id?

Looking at the response that you got makes me think that its expecting everything to be wrapped in something and that is why it did not find the 2 required fields.

@camerondavison, the “datetime” requires adhering to ISO8601 format as per JSON schema specification. The corresponding JSON schema is here.