Lamda Function For PageView

[lalspicesresearch]

Hello,
I have setup collector and enricher successfully.I have successfully triggered PageView event of page and i have used data delivery firehose to transfer data from data stream to S3 bucket. But now i want to get the PageView data in JSON format so for that i have to use Lamda Function. But can you suggest me lamda function for PageView event data?

Can anyone tell is the data populated in S3 bucket is enriched data or not?

d
27.34.25.1
�uC�x��UTF-8�ssc-1.0.0-kinesis,hMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.366http://127.0.0.1:8282/@#/com.snowplowanalytics.snowplow/tp2T}{“schema”:“iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4”,“data”:[{“e”:“pv”,“url”:“http://127.0.0.1:8282/",“page”:“Laravel”,“tv”:“js-2.15.0”,“tna”:“cfZanui”,“aid”:“zanuihostjsappid”,“p”:“web”,“tz”:“Asia/Kathmandu”,“lang”:“en-US”,“cs”:“UTF-8”,“f_pdf”:“1”,“f_qt”:“0”,“f_realp”:“0”,“f_wma”:“0”,“f_dir”:“0”,“f_fla”:“0”,“f_java”:“0”,“f_gears”:“0”,“f_ag”:“0”,“res”:“1366x768”,“cd”:“24”,“cookie”:“1”,“eid”:“baa9d2d5-43fd-42d8-8dbd-92b5294c65cb”,“dtm”:“1603160405405”,“vp”:“1294x600”,“ds”:“1294x600”,“vid”:“1”,“sid”:“18ebede0-9633-488f-abfd-9050f6c69dfa”,“duid”:“23722df0-a95d-48d5-b2c7-d87d9ddca412”,“stm”:"1603160405409”}]}^
eTimeout-Access: Host: 13.54.178.43:8181Connection: keep-alivetUser-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36Accept: */*Origin: http://127.0.0.1:8282Referer: http://127.0.0.1:8282/Accept-Encoding: gzip, deflate Accept-Language: en-US, en;q=0.9application/jsonhapplication/json�13.54.178.43�$9660d2ad-dddd-4acc-834d-691bf90d0605ziAiglu:com.snowplowanalytics.snowplow/CollectorPayload/thrift/1-0-0zanuihostjsappid web 2020-10-20 02:20:12.415 2020-10-20 02:20:06.175 2020-10-20 02:20:05.405 page_view baa9d2d5-43fd-42d8-8dbd-92b5294c65cb cfZanui js-2.15.0 ssc-1.0.0-kinesis stream-enrich-1.0.0-common-1.0.0 27.34.25.1 23722df0-a95d-48d5-b2c7-d87d9ddca412 1 9660d2ad-dddd-4acc-834d-691bf90d0605 http://127.0.0.1:8282/ Laravel http 127.0.0.1 8282 / Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36 en-US 1 0 0 0 0 0 0 0 0 1 24 1294 600 Asia/Kathmandu 1366 768 UTF-8 1294 600 2020-10-20 02:20:05.409 18ebede0-9633-488f-abfd-9050f6c69dfa 2020-10-20 02:20:06.171 com.snowplowanalytics.snowplow page_view jsonschema 1-0-0

Hope to get help from the community.
Thank you

[mike]

The event you’ve posted looks like a raw event - rather than an enriched event produced by stream-enrich. Once you run this event through the stream enrichment process you can then use a Lambda function (and one of the analytics SDKs) to convert this event from a TSV representation to a JSON representation.

[lalspicesresearch]

Do i need to write Lamda function myself or there will be built-in Lamda function in AWS to convert TSV representation to JSON?
Thank You

[mike]

Yes - you will need to write the Lambda function yourself, but you can use one of the Snowplow Analytics SDKs to do the conversion.

[niksik]

According to your AWS setup Collector -> KDS -> Enrich -> KDS -> Firehose -> S3, normally this event has passed the enrichment process so to my understanding it’s considered enriched!
On the other hand, the data in a S3 are a string representation of a data stream meaning that you need to apply a simple transformation using Lamda within Firehose that adds a new line (\n) after each event of the stream! The produced files will be then TSV and also separated by new line, a suitable format in order to be saved directly to Redshift.

[lalspicesresearch]

@mike @niksik i have made uploaded zip in lambda function where i have included dependecies i.e snowplow-analytics-sdk but why it is giving error?

Thank you!