Managing secrets in CI/CD for AI and LLM workloads

AI workloads multiply secrets because each pipeline juggles model provider keys, vector database credentials, and fine-tune storage. Manage them safely by storing values in your CI provider’s secret store, preferring OIDC short-lived tokens over static keys, scoping keys per environment, masking logs, scanning for leaks, and rotating on a schedule. The principles are old, but the volume of credentials in an AI build is what makes discipline essential.

TL;DR:

• AI pipelines carry more secrets: model provider API keys, vector DB creds, object-store keys for fine-tune data, and observability tokens, all in one build.
• Store values in the CI secret store, never in code, env files, or container layers.
• Prefer OIDC short-lived tokens over static keys, per GitHub’s documentation.
• Scope per environment, mask logs, scan for leaks, and rotate. If a key has already leaked, see committed a secret to GitHub.

Why do AI workloads multiply secrets?

A standard web app build might need one database URL and a deploy token. An AI workload needs far more. A single inference or fine-tune pipeline commonly authenticates to one or more model providers (OpenAI, Anthropic, or a hosted open-weights endpoint), a vector database (Pinecone, Weaviate, or pgvector behind a managed Postgres), an object store holding training and fine-tune data, and an observability or eval platform. Each of those is a separate credential.

That spread is the problem. More keys, held in more workflow files and more environments, means a wider surface for one of them to leak. The dangers are familiar but amplified:

• Keys in logs: an echo or a verbose SDK debug line prints a provider key into a build log anyone with read access can see.
• Keys in build artifacts: a .env file copied into a tarball or a notebook output cell can carry a live key downstream.
• Keys baked into container images: a key set during docker build persists in an image layer even if a later layer unsets it.
• Long-lived keys: a model provider key with no expiry works until someone rotates it, so a single exposure can be exploited for months and run up real inference spend.

How do you manage secrets step by step?

The workflow below applies to any AI build. The examples use GitHub Actions, but the shape carries to GitLab CI, CircleCI, or others.

1. Put every value in the CI provider’s secret store. In GitHub, use repository or environment secrets, never plaintext in the workflow YAML. Reference them as ${{ secrets.OPENAI_API_KEY }}.
2. Prefer OIDC short-lived tokens over static keys. For cloud access (the object store holding fine-tune data, for example), authenticate with OIDC rather than a stored access key. See GitHub Actions OIDC without secrets for the full setup.
3. Scope keys per environment. Use separate keys for development, staging, and production via GitHub environment secrets, so a leaked staging key cannot touch production data.
4. Mask anything secret-shaped in logs. GitHub redacts stored secrets automatically; mask anything else with ::add-mask::.
5. Scan for leaks before merge and on a schedule. Run a secret scanner in CI and as a pre-commit hook. See gitleaks vs trufflehog.
6. Rotate on a schedule and after any exposure. OIDC tokens rotate themselves; for unavoidable static keys, set a reminder and rotate per environment.

What does a safe GitHub Actions job look like?

The secret is injected as an environment variable scoped to the single step that needs it, and is never echoed:

name: build-rag-index
on:
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  index:
    runs-on: ubuntu-latest
    environment: production   # scopes which secrets are available
    steps:
      - uses: actions/checkout@v4
      - name: Embed and upsert
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
        run: python build_index.py   # reads keys from env, never prints them

For a value computed at runtime that you want hidden from logs, mask it. Per GitHub’s encrypted secrets documentation, ::add-mask:: causes a value to be treated as a secret and redacted:

      - name: Mask a derived token
        run: |
          TOKEN=$(./mint-finetune-token.sh)
          echo "::add-mask::$TOKEN"
          echo "FINETUNE_TOKEN=$TOKEN" >> "$GITHUB_ENV"

Which risks does each control address?

Risk	Primary control	How it helps
Key printed in build log	Secret store plus ::add-mask::	Stored secrets are auto-redacted; masking covers derived values
Static key exploited for months	OIDC short-lived tokens	Token expires when the job ends, so there is nothing durable to steal
Staging leak hits production	Per-environment secrets	A scoped key cannot reach another environment’s data or spend
Key baked into an image layer	Build-time mounts, not ENV/ARG	The key never persists in the published image
Fork reads your keys	CI default fork policy	Per GitHub’s documentation, secrets are not passed to forked-repo runs except GITHUB_TOKEN
Committed key in history	Leak scanning plus rotation	A scanner catches it; rotation makes the exposed value useless

Why prefer OIDC over static keys?

A stored model-store or cloud key is a standing liability: it works until rotated, and anyone who can read, leak, or print it can use it from anywhere. Per GitHub’s OpenID Connect documentation, OIDC lets a workflow request a short-lived access token directly from the cloud provider, unique to that run and expiring when the job completes. There is nothing static to steal.

OpenID Connect is an identity layer on top of OAuth 2.0. Per the OpenID Foundation, it answers “who are you?” with a signed token rather than handing over a reusable password. Apply it wherever a provider supports it, and reserve stored secrets for the providers (still common among AI vendors) that only issue static API keys. For those, treat the OWASP secrets management guidance as your baseline: centralise storage, scope access, and rotate.

Where to go next

Move cloud and object-store access onto GitHub Actions OIDC without secrets so fine-tune data keys stop living in the secret store. Add a scanner from gitleaks vs trufflehog to your pre-commit and CI gates. And if a provider key has already slipped into a commit, follow committed a secret to GitHub: rotate first, scrub history second. Browse more in the guides library.