Running iglu-server (schema repo) locally for snowplow-micro

Hey,

we’re running snowplow-micro to test and validate our schemas. For this purpose, we created 4 docker containers:

    ├── iglu-server
    │   ├── Dockerfile
    │   └── iglu_conf.conf
    ├── igluctl
    │   ├── Dockerfile
    │   └── schemas
    │       └── com.myapp
    │           └── minimal_tracking_event
    │               └── jsonschema
    │                   └── 1-0-0
    ├── postgres
    │   ├── Dockerfile
    │   └── init.sql
    └── sp-micro
        ├── Dockerfile
        ├── iglu.json
        └── micro_conf.conf

In the igluctl container we’re trying to push our schemas to our local iglu server.

FROM snowplow-docker-registry.bintray.io/snowplow/igluctl:0.6.0

COPY schemas/ ./schemas

CMD ["static", "push", "--public", "./schemas", "iglumock:80/iglu-server", "4ab49682-d0d5-11ea-87d0-0242ac130003"]

Linting the schemas in this container worked without a problem but pushing it to the iglu-repo results in the following error:

No usable value for read
Did not find value which can be converted into java.lang.String

For our iglu-server we use the following config:

repo-server {
  interface = "0.0.0.0"
  port = 8080
  pool = "cached"
}

# 'postgres' contains configuration options for the postgre instance the server is using
# 'dummy' is in-memory only storage
database {
  type = "postgres"
  host = "postgres"
  port = 5432
  dbname = "igludb"
  username = "postgres"
  password = "iglusecret"
  driver = "org.postgresql.Driver"
  maxPoolSize = 5
  pool = {
  }
}

# Enable additional debug endpoint to output all internal state
debug = true

And the iglu-server has an interface at:

[2020-07-29T08:15:26.313Z] [ioapp-pool-1-thread-1] INFO org.http4s.server.blaze.BlazeServerBuilder - http4s v0.20.19 on blaze v0.14.11 started at http://0.0.0.0:8080/

Maybe our approach of setting up snowplow-micro is too complicated but it would be good to know whether we use the igluctl static push command correctly or whether something about our iglu-server config is incorrect.

1 Like

Hi @mgloel,

You don’t need to have a new image to run igluctl subcommands or iglu server. I’d recommend using official docker images of iglu server and igluctl, which allows using subcommands as if you’re using the executable.

Please have a look at a sample run. In your scenario, you need the following to start with:

$ docker run snowplow-docker-registry.bintray.io/snowplow/igluctl:0.6.0 lint

It’ll show you expected parameters and flags with their meaning.

Please also check

$ docker run snowplow-docker-registry.bintray.io/snowplow/igluctl:0.6.0 static push

to see usage, options and flags with their meanings.

To run Iglu Server with Postgres, please have a look at our example running Iglu Server and Postgres together in a docker-compose example where README shows how to push your schemas to Iglu Server.

I believe using official images will ease your workflow. Please let us know how it goes!

2 Likes

Hi @mgloel,

I can suggest a much simpler setup of micro and iglu, which might help you out.

The trick is to create a iglu.json file that fetches schemas from local files instead of from a http server. My iglu.json file looks like this:

{
  "schema": "iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-0",
    "data": {
      "cacheSize": 500,
      "repositories": [
        {
          "name": "Iglu Central",
          "priority": 0,
          "vendorPrefixes": [ "com.snowplowanalytics" ],
          "connection": {
            "http": {
              "uri": "http://iglucentral.com"
            }
          }
        },
        {
          "name": "local Iglu repository",
          "priority": 5,
          "vendorPrefixes": [ "test.example.iglu" ],
          "connection": {
            "http": {
              "uri": "file:///local-iglu"
            }
          }
        }
      ]
    }
}

Notice the line "uri": "file:///local-iglu"

Finally, you need to mount the schemas into the micro docker container when you run it:

docker run \                                                                                                                                                                          
  --mount type=bind,source=$PWD/example,destination=/config \                                                                                                                
  --mount type=bind,source=$PWD/schemas,destination=/local-iglu \                                                
  -p 9090:9090 \                                                                                   
  snowplow/snowplow-micro:latest --collector-config /config/micro.conf --iglu /config/iglu.json

With this setup you don’t need any of the iglu-server, igluctl or postgres docker containers in order for micro to see your schemas.