Skip to content

Ingest Your Catalog with Ocean ​

The demo catalog (repositories, GitLab projects, GCP resources, AWS resources) is filled automatically by Port Ocean exporters running on a schedule in GitHub Actions. The IDP implements the Port-compatible API, so standard Ocean integrations work against it out of the box.

After this page you'll have the same setup as Admin → Data sources in the demo:

Data sources with four Ocean exporters

1. Create a service account ​

Ocean authenticates with a service account (client ID + secret), not a user account.

  1. Go to Admin → Credentials → Create Service Account (the demo uses ocean-ingest).
  2. Copy the Client ID (pa_...) and Client Secret — the secret is shown only once.

Service accounts under Admin → Credentials

  1. Grant it a role. Service accounts have no permissions by default — Ocean will fail with 403 on /v1/integration/provision-enabled if you skip this. Go to Admin → RBAC → Grant access and assign the service account an organization role (the demo grants Organization Owner):

RBAC page showing the ocean-ingest service account with Organization Owner

Permission changes are cached

RBAC deny decisions are cached for ~5 minutes — if you just granted the role and still get 403s, wait a few minutes.

2. Add the ingestion workflow to a config repository ​

The demo keeps its ingestion pipeline in idpnextdemo/deployments-config-v2. Create .github/workflows/ingest-idpnext.yml:

yaml
name: ingest-idpnext
on:
  workflow_dispatch:
  schedule:
    - cron: "0 6 * * *"

env:
  IDP_BASE_URL: https://idpnext.wazyneo.com

jobs:
  ingest:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - name: GitHub (idpnextdemo org)
        continue-on-error: true
        uses: port-labs/ocean-sail@v1
        with:
          type: github-ocean
          port_client_id: ${{ secrets.IDP_CLIENT_ID }}
          port_client_secret: ${{ secrets.IDP_CLIENT_SECRET }}
          port_base_url: ${{ env.IDP_BASE_URL }}
          version: "4.5.6-beta"
          config: |
            githubHost: https://api.github.com
            github_host: https://api.github.com
            githubToken: "${{ secrets.PAT_GITHUB_TOKEN }}"
            github_token: "${{ secrets.PAT_GITHUB_TOKEN }}"
            githubOrganization: idpnextdemo
            github_organization: idpnextdemo

      - name: GitLab (idpnextdemo group)
        continue-on-error: true
        uses: port-labs/ocean-sail@v1
        with:
          type: gitlab-v2
          port_client_id: ${{ secrets.IDP_CLIENT_ID }}
          port_client_secret: ${{ secrets.IDP_CLIENT_SECRET }}
          port_base_url: ${{ env.IDP_BASE_URL }}
          config: |
            gitlabToken: "${{ secrets.PAT_GITLAB_TOKEN }}"
            gitlab_token: "${{ secrets.PAT_GITLAB_TOKEN }}"

      - name: GCP (idpnextdemo project)
        continue-on-error: true
        uses: port-labs/ocean-sail@v1
        with:
          type: gcp
          port_client_id: ${{ secrets.IDP_CLIENT_ID }}
          port_client_secret: ${{ secrets.IDP_CLIENT_SECRET }}
          port_base_url: ${{ env.IDP_BASE_URL }}
          config: |
            encoded_adc_configuration: ${{ secrets.ENCODED_ADC_IDPNEXTDEMO }}

      - name: AWS
        continue-on-error: true
        uses: port-labs/ocean-sail@v1
        with:
          type: aws
          port_client_id: ${{ secrets.IDP_CLIENT_ID }}
          port_client_secret: ${{ secrets.IDP_CLIENT_SECRET }}
          port_base_url: ${{ env.IDP_BASE_URL }}
          config: |
            aws_access_key_id: ${{ secrets.AWSACCESSKEYID }}
            aws_secret_access_key: ${{ secrets.AWSSECRETACCESSKEY }}

Add the repository secrets it references:

SecretValue
IDP_CLIENT_ID / IDP_CLIENT_SECRETthe service account credentials from step 1
PAT_GITHUB_TOKENa GitHub token that can read your org's repos/teams/users
PAT_GITLAB_TOKENa GitLab token with api scope on your group
ENCODED_ADC_IDPNEXTDEMObase64-encoded GCP service account key (viewer roles)
AWSACCESSKEYID / AWSSECRETACCESSKEYread-only AWS credentials

Only add the blocks/secrets for the providers you actually use — each step is continue-on-error, so one failing provider doesn't block the others.

3. Run it ​

Trigger the workflow manually the first time (Actions → ingest-idpnext → Run workflow). Each exporter:

  1. Authenticates against the IDP with the service account.
  2. Provisions its default blueprints and integration config (visible in Admin → Data sources).
  3. Syncs entities into the catalog.

From then on it runs daily at 06:00 UTC (the cron trigger).

4. Scope and customize the mappings ​

Each data source has a mapping: rules that transform provider API payloads into entities using jq expressions. Mappings live in the IDP (IntegrationConfig) and are editable via API — no workflow change needed:

bash
curl -X PATCH "https://<your-idp-domain>/v1/integration/my-gitlab-v2-integration" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d @mapping.json

Example — the demo's GitLab project mapping, which maps GitLab projects into the service blueprint and scopes the sync to one group via the selector query:

yaml
resources:
  - kind: project
    selector:
      query: '.path_with_namespace | startswith("idpnextdemo/")'
      includedFiles:
        - README.md
        - CODEOWNERS
      includeLanguages: true
    port:
      entity:
        mappings:
          identifier: '.path_with_namespace | gsub(" "; "")'
          title: .name
          blueprint: '"service"'
          properties:
            url: .web_url
            readme: '.__includedFiles["README.md"]'
            language: '.__languages // {} | to_entries | max_by(.value) | .key'
            codeowners: '.__includedFiles["CODEOWNERS"]'

And the GitHub repository mapping:

yaml
resources:
  - kind: repository
    selector:
      query: 'true'
    port:
      entity:
        mappings:
          identifier: .name
          title: .name
          blueprint: '"githubRepository"'
          properties:
            url: .html_url
            readme: file://README.md
            language: if .language then .language else "" end
            visibility: if .private then "private" else "public" end
            defaultBranch: .default_branch
          relations:
            organization: .owner.login

Tightening a filter doesn't delete old entities

The IDP does not implement Ocean's stale-entity deletion. If you narrow a selector after a sync, entities that no longer match stay in the catalog — delete them manually via the UI or API.

Test a mapping before you save it ​

You don't have to wait for the next sync to see what a jq change produces. Open the data source from Admin → Data sources and use the Mapping tab:

Testing a mapping against real samples

  1. Kinds to test lists every kind the source has sent samples for (the ingest workflow stores the last few raw payloads per kind — a green dot means samples exist; the refresh icon reloads them). Pick a kind, then choose one of its samples from the dropdown.
  2. The sample editor shows the raw provider payload. It is editable: tweak a field, or paste any JSON you want to try the mapping against.
  3. Edit the YAML in Mapping, then click Test mapping. The Test result panel shows, per kind, the entities the mapping would produce — identifier, title, blueprint, properties and relations — exactly as the sync would write them. A mapping error for one kind is reported under that kind without hiding the others.

Nothing is written to the catalog by a test. When the result looks right, Resync to apply the saved mapping to the whole source.

No samples yet?

Samples arrive with the first ingest run. Until then, paste a representative payload from the provider's API into the sample editor and test against that.

5. Security alerts (Dependabot, code scanning, secret scanning) ​

The GitHub exporter can also sync a repository's security alerts into the catalog. Add one blueprint per alert type (the demo uses githubDependabotAlert, githubCodeScanningAlert and githubSecretScanningAlert, each with a repository relation to githubRepository) and one resources[] entry per kind to the GitHub mapping:

yaml
resources:
  - kind: dependabot-alert
    selector:
      query: 'true'
      states: [open, dismissed, fixed, auto_dismissed]   # sync fixed alerts too, so they turn green instead of staying stale
      # severity: [critical, high]      # optional
      # ecosystems: [npm, pip]          # optional
    port:
      entity:
        mappings:
          identifier: '.__repository + "-" + (.number | tostring)'
          title: .security_advisory.summary
          blueprint: '"githubDependabotAlert"'
          properties:
            state: .state
            severity: .security_vulnerability.severity // .security_advisory.severity
            cvssScore: .security_advisory.cvss.score
            ghsaId: .security_advisory.ghsa_id
            cveId: .security_advisory.cve_id
            packageName: .dependency.package.name
            ecosystem: .dependency.package.ecosystem
            manifestPath: .dependency.manifest_path
            firstPatchedVersion: .security_vulnerability.first_patched_version.identifier
            url: .html_url
            createdAt: .created_at
            fixedAt: .fixed_at
          relations:
            repository: .__repository

  - kind: code-scanning-alerts          # note the plural
    selector:
      query: 'true'
      state: open                        # a single state here (open | closed | dismissed | fixed)
    port:
      entity:
        mappings:
          identifier: '.__repository + "-" + (.number | tostring)'
          title: .rule.description // .rule.name // .rule.id
          blueprint: '"githubCodeScanningAlert"'
          properties:
            state: .state
            severity: .rule.security_severity_level // .rule.severity
            ruleId: .rule.id
            tool: .tool.name
            path: .most_recent_instance.location.path
            startLine: .most_recent_instance.location.start_line
            url: .html_url
            createdAt: .created_at
          relations:
            repository: .__repository

  - kind: secret-scanning-alerts        # note the plural
    selector:
      query: 'true'
      state: all                         # open | resolved | all
      hideSecret: true                   # keep it — the secret value then never leaves GitHub
    port:
      entity:
        mappings:
          identifier: '.__repository + "-" + (.number | tostring)'
          title: '(.secret_type_display_name // .secret_type) + " #" + (.number | tostring)'
          blueprint: '"githubSecretScanningAlert"'
          properties:
            state: .state
            secretType: .secret_type
            validity: .validity
            publiclyLeaked: .publicly_leaked
            resolution: .resolution
            url: .html_url
            createdAt: .created_at
          relations:
            repository: .__repository

Every alert payload is enriched by the exporter with __repository (the bare repository name) and __organization, which is what the identifiers and the repository relation are built from — alert numbers are only unique per repository.

Token permissions

Alerts need the classic repo scope on the token (or, fine-grained: Dependabot alerts, Code scanning alerts and Secret scanning alerts read). Without it each kind logs a 403 and syncs nothing. You also only get alerts for repositories where the feature is enabled (Settings → Code security on GitHub) — code scanning alerts exist only once a CodeQL/SARIF workflow has run. A 403 for one repository is logged as a warning and does not stop the sync.

The demo shows them under Catalog → Github → Security, with a Security Overview dashboard as the folder's first page.

6. Ingest a repository from another organization ​

The GitHub exporter reads one organization (githubOrganization). To bring in a repository you don't own — the demo adds the public angular/angular — run a second exporter against that organization, as a separate data source (its own port_client_id... block in the workflow with a distinct integration identifier, e.g. angular-github-ocean), and narrow every kind to the one repository with repoSearch:

yaml
resources:
  - kind: repository
    selector:
      query: 'true'
      repoSearch:
        query: 'angular in:name stars:>80000'
    port:
      entity:
        mappings:
          identifier: '.owner.login + "-" + .name'     # angular-angular
          title: .full_name                             # angular/angular
          blueprint: '"githubRepository"'
          properties:
            url: .html_url
            readme: file://README.md
            defaultBranch: .default_branch
  - kind: pull-request
    selector:
      query: 'true'
      repoSearch: { query: 'angular in:name stars:>80000' }
      states: [open, closed]
      since: 30                # closed PRs updated in the last N days (max 90)
      maxResults: 300
    port:
      entity:
        mappings:
          identifier: '.__organization + "-" + .__repository + "-pr-" + (.number | tostring)'
          title: .title
          blueprint: '"githubPullRequest"'
          properties:
            status: if .merged_at then "merged" else .state end
            prNumber: .number
            creator: .user.login
            createdAt: .created_at
            mergedAt: .merged_at
          relations:
            repository: '.__organization + "-" + .__repository'
  - kind: branch
    selector:
      query: 'true'
      repoSearch: { query: 'angular in:name stars:>80000' }
      detailed: true           # one extra request per branch: head commit author/date/message
    port:
      entity:
        mappings:
          identifier: '.__organization + "-" + .__repository + "-branch-" + (.name | gsub("/"; "-"))'
          title: .name
          blueprint: '"githubBranch"'
          properties:
            isDefault: .name == .__repository_object.default_branch
            protected: .protected
            commitAuthor: .commit.author.login // .commit.commit.author.name
            commitDate: .commit.commit.committer.date
          relations:
            repository: '.__organization + "-" + .__repository'
  - kind: issue
    selector:
      query: '.pull_request == null'   # the issues API also returns pull requests
      repoSearch: { query: 'angular in:name stars:>80000' }
      state: open
    port:
      entity:
        mappings:
          identifier: '.__organization + "-" + .__repository + "-issue-" + (.number | tostring)'
          title: .title
          blueprint: '"githubIssue"'
          properties:
            state: .state
            labels: '[.labels[].name]'
            comments: .comments
            createdAt: .created_at
          relations:
            repository: '.__organization + "-" + .__repository'

Prefix the identifiers with the owner (angular-angular, angular-angular-pr-70723, …) so they never collide with a fork of the same repository synced by your main data source.

repoSearch matches more than you think

The exporter runs GET /search/repositories?q=org:<org> <your query>, and GitHub search ORs location qualifiers — so repoSearch.query: 'repo:angular/angular' matches the whole angular organization (200+ repositories, thousands of pull requests and branches, and the 5 000 requests/hour token budget is gone in one run). Use qualifiers that single out the repository instead, such as <name> in:name stars:>N or <name> in:name created:YYYY-MM-DD, and dry-run the query against the search API before the first sync.

Security alerts on a repository you don't administer

GitHub only exposes Dependabot / code scanning / secret scanning alerts to collaborators with security access — a token gets 403 on angular/angular. Keep those kinds for your own repositories (or a fork you administer).

Next: connect GitLab for pipelines →

IDP Next — Internal Developer Platform