Tutorial - Job-Based Fire and Forget Mode

Get in Touch

If you need assistance or have questions, we are here to help. Please reach out to us through our company website at www.dna-evolutions.com/contact.

Overview

• Introduction — Classic FAF vs. Job Mode
• How Job Mode Works
• The JobAcceptedResponse
• Tenant Isolation and Security
• Endpoint Reference
• The DatabaseJobItemSearch and DatabaseJobInfoSearch Models
• Step-by-Step: Running a Job-Mode Optimization
• Encryption at Rest
• KMS Envelope Encryption
• Encryption Mode Summary
• What Gets Stored in MongoDB
• Error Handling

Introduction

The classic Fire and Forget (FAF) mode lets you submit a long-running optimization without keeping the HTTP connection open. The server returns a boolean "started" signal, and you later search for results by creator name using the persistence-read endpoints.

Job Mode builds on this foundation and adds a structured, tenant-aware job lifecycle. Instead of a boolean, the server returns a JobAcceptedResponse containing a unique jobId. This single token is everything the client needs to track and retrieve the job — no searching by creator, no guessing which result belongs to which submission.

When to use which mode

Job Mode does not replace classic FAF. Both modes coexist and share the same optimization engine, database, and persistence settings. Job Mode simply adds a structured job handle and enforces tenant scoping.

How Job Mode Works

The lifecycle of a job-mode optimization follows four phases:

1. Submit

The client sends a POST request to /api/job/optimizefaf/runFAF with the standard RestOptimization body and an X-Tenant-Id header (injected by the API gateway). The server generates a jobId, starts the optimization asynchronously, and immediately returns an HTTP 202 with the JobAcceptedResponse.

2. Poll

While the optimization is running, the client polls for progress, status, warnings, or errors using the job-based read endpoints under /api/db/job/read/. All requests require the jobId and tenantId via a DatabaseJobInfoSearch body — the server only returns data that belongs to the authenticated tenant.

3. Retrieve

Once the optimization completes (the status shows SUCCESS_WITH_SOLUTION or SUCCESS_WITHOUT_SOLUTION), the client retrieves the full result via /api/db/job/read/findOptimization using a DatabaseJobItemSearch body. If the data was encrypted with a client secret during submission, the same secret must be provided in this request body to decrypt the result. If KMS encryption was used by the server, decryption happens transparently — no secret is needed from the client.

4. Expire

Like classic FAF, persisted data has a configurable TTL (expiry in the persistenceSetting). After expiry, the data is automatically cleaned up by the scheduler.

The JobAcceptedResponse

When you submit a job, the server responds with HTTP 202 and the following body:

{
  "jobId": "648d2724-3a77-47f4-b937-d3ab6abf2341",
  "creatorHash": "11aa65b13c2a6d34f8727e82e403ce869e3bba1d35c45c595e8cc5ce5e74e57a",
  "ident": "JOpt-Run-1774127074120",
  "submittedAt": 1774131229940,
  "status": "ACCEPTED"
}

Tenant Isolation and Security

Job Mode enforces tenant isolation at the API level. Every persisted document (result, progress, status, warning, error) is tagged with both a jobId and a tenantId. All read queries are automatically scoped by both fields — a client can only access data that matches its authenticated tenant.

How the tenantId flows

1. The client authenticates with the API gateway (e.g. Azure API Management) via an API key or OAuth token.
2. The gateway resolves the subscription to a tenant and injects the X-Tenant-Id header.
3. TourOptimizer reads the header server-side — the client cannot forge it.
4. All persistence writes tag the data with this tenantId.
5. All persistence reads filter by tenantId — even if someone knows another tenant's jobId, the query returns nothing.

The jobId is not a security token

The jobId is a convenience handle, not an access credential. Security is enforced by the tenantId (which comes from the verified gateway header), not by the secrecy of the jobId. The jobId is a UUID v4 with 122 bits of randomness, which makes it practically unguessable, but this is defense-in-depth — not the primary boundary.

Endpoint Reference

All job-mode endpoints are under the /api/job/ path prefix and require database mode to be enabled (DNA_DATABASE_ACTIVE=true).

Submit

Required headers: X-Tenant-Id

Poll and Retrieve

The DatabaseJobItemSearch and DatabaseJobInfoSearch Models

Job-mode read endpoints use two request body models. Both require jobId and tenantId as mandatory fields.

DatabaseJobItemSearch

Used by findOptimization to retrieve the full optimization result. Includes an optional secret field for decrypting client-encrypted results. For KMS-encrypted or unencrypted results, the secret field can be omitted.

{
  "jobId": "648d2724-3a77-47f4-b937-d3ab6abf2341",
  "tenantId": "my-tenant-123",
  "secret": "",
  "timeOut": "PT1M"
}

DatabaseJobInfoSearch

Used by findProgress, findStatus, findWarning, and findError to retrieve stream data.

{
  "jobId": "648d2724-3a77-47f4-b937-d3ab6abf2341",
  "tenantId": "my-tenant-123",
  "limit": 10,
  "sortDirection": "DESC",
  "timeOut": "PT1M"
}

Step-by-Step: Running a Job-Mode Optimization

This walkthrough assumes you have already set up MongoDB and TourOptimizer as described in the classic FAF tutorial (Steps 1–4).

Step 1: Submit the job

Send a POST to /api/job/optimizefaf/runFAF with your optimization input. Include the X-Tenant-Id header and the persistenceSetting in the extension.

Request:

POST /api/job/optimizefaf/runFAF
Content-Type: application/json
X-Tenant-Id: my-tenant-123

The request body is the standard RestOptimization JSON — identical to classic FAF. Make sure enablePersistence is set to true inside the persistenceSetting.

The secret field determines which encryption mode is used on the server:

• secret is non-empty → CLIENT mode (you manage the key)
• secret is empty + KMS enabled on server → KMS mode (server manages the key, transparent to you)
• secret is empty + no KMS → no encryption

Response (HTTP 202):

{
  "jobId": "648d2724-3a77-47f4-b937-d3ab6abf2341",
  "creatorHash": "11aa65b13c2a6d34f8727e82e403ce869e3bba1d35c45c595e8cc5ce5e74e57a",
  "ident": "JOpt-Run-1774127074120",
  "submittedAt": 1774131229940,
  "status": "ACCEPTED"
}

Save the jobId — you will need it for all subsequent requests.

Step 2: Poll for progress (optional)

While the optimization is running, you can check progress:

POST /api/db/job/read/findProgress
Content-Type: application/json

{
  "jobId": "648d2724-3a77-47f4-b937-d3ab6abf2341",
  "tenantId": "my-tenant-123"
}

The response is a stream of JOptOptimizationProgress objects showing the current stage and percentage.

Step 3: Check status

To determine whether the optimization has finished:

POST /api/db/job/read/findStatus
Content-Type: application/json

{
  "jobId": "648d2724-3a77-47f4-b937-d3ab6abf2341",
  "tenantId": "my-tenant-123"
}

Look for a status with statusDescription equal to SUCCESS_WITH_SOLUTION or ERROR.

Step 4: Retrieve the result (no client secret — covers both unencrypted and KMS)

If you did not provide a secret during submission, retrieve the result without one. This works regardless of whether the server applied KMS encryption or stored the data unencrypted — the server detects the encryption mode from the stored metadata and decrypts transparently.

POST /api/db/job/read/findOptimization
Content-Type: application/json

{
  "jobId": "648d2724-3a77-47f4-b937-d3ab6abf2341",
  "tenantId": "my-tenant-123",
  "timeOut": "PT1M"
}

The response contains the full RestOptimization object with the computed solution, route assignments, scheduling details, and violation reports.

Step 5: Retrieve the result (with client secret — CLIENT mode only)

If a non-empty secret was provided in the persistenceSetting during submission, you must include the same secret in the DatabaseJobItemSearch request body:

{
  "jobId": "648d2724-3a77-47f4-b937-d3ab6abf2341",
  "tenantId": "my-tenant-123",
  "secret": "YourStr0ng!Secret_Here",
  "timeOut": "PT1M"
}

The server uses the secret together with the stored salt and IV to re-derive the AES key and decrypt the result. If the secret is missing, incorrect, or does not match the one used during submission, the server returns an error indicating that the file is corrupt or the secret was not provided.

Encryption at Rest

Job Mode supports three encryption modes for the optimization result payload. The mode is determined automatically at submission time based on a simple priority rule.

Encryption mode priority

1. Client secret is non-empty → CLIENT mode (client manages the key, KMS is ignored even if enabled)
2. No client secret + KMS is enabled for the tenant → KMS mode (server manages the key transparently)
3. No client secret + no KMS → No encryption (data is compressed only)

The secret field in persistenceSetting.mongoSettings is the switch: provide a passphrase and you control the key; leave it empty and the server decides based on its KMS configuration.

CLIENT mode — Client-Managed Secret

This is the explicit encryption mode where the client provides and manages the passphrase.

During submission:

1. The client includes a non-empty secret in the persistenceSetting.mongoSettings.secret field.
2. The server validates the secret strength. If it is too weak, the job is rejected with HTTP 400.
3. A random 16-byte salt and 12-byte IV are generated.
4. A 256-bit AES key is derived from the secret using PBKDF2 (salt, 310,000 iterations).
5. The optimization result is serialized, compressed with bzip2, and encrypted with AES-256-GCM using the derived key and IV.
6. The encrypted bytes are stored in GridFS.
7. The IV, salt, algorithm names, iteration count, and key length are stored in the GridFS metadata sec block with encMode: "CLIENT".
8. The secret is discarded from server memory. It is never persisted.

During retrieval:

1. The client provides the same secret in the DatabaseJobItemSearch.secret field.
2. The server reads the sec block from the GridFS metadata and sees encMode: "CLIENT".
3. The AES key is re-derived using the stored salt, iteration count, and the provided secret.
4. The ciphertext is decrypted with AES-256-GCM using the re-derived key and the stored IV.
5. If the authentication tag does not match (wrong secret, tampered data), decryption fails and the server returns an error.
6. On success, the decrypted bytes are decompressed (bzip2) and deserialized into the RestOptimization response.

If you lose the secret: The data cannot be recovered. There is no server-side copy of the secret, no master key, and no administrative backdoor. This is by design — it guarantees that only someone who knows the original passphrase can read the optimization result. Store your secrets in a secure location (e.g. a secrets manager or password vault).

Secret strength validation

When a non-empty secret is provided, the server validates it before accepting the job. The secret must meet minimum requirements for length and character class diversity (mixing uppercase, lowercase, digits, and special characters). If the secret is too weak, the submission is rejected immediately with HTTP 400 Bad Request — the optimization is never started.

CLIENT mode algorithm details

Why AES-GCM? GCM provides authenticated encryption — if anyone tampers with the encrypted data in MongoDB, decryption fails with an authentication error rather than silently producing corrupted output.

Why PBKDF2? The client provides a human-chosen passphrase, not a raw cryptographic key. PBKDF2 stretches this passphrase through 310,000 iterations of HMAC-SHA256 to produce a 256-bit AES key, making brute-force attacks computationally expensive.

Why random IV and salt per job? Even if two different jobs use the same passphrase, the randomly generated salt produces a different derived key, and the randomly generated IV ensures different ciphertext. This prevents pattern analysis across jobs.

KMS Envelope Encryption

When the server has a KMS (Key Management Service) configured and the client does not provide a secret, the server automatically encrypts the optimization result using envelope encryption. This is completely transparent to the client — the API request and response look identical to an unencrypted job.

How KMS mode works

Instead of deriving a key from a client passphrase, the server generates a random 256-bit AES key (the data encryption key, or DEK) for each job. The result is encrypted with this DEK. The DEK itself is then encrypted ("wrapped") by a key encryption key (KEK) managed in an external KMS (e.g. Azure Key Vault). The wrapped DEK is stored in the GridFS metadata. The plaintext DEK is discarded from memory.

On retrieval, the server reads the wrapped DEK from the metadata, unwraps it via the KMS, decrypts the result, and returns it. The client never sees the DEK or interacts with the KMS.

KMS mode end-to-end flow

During submission (encryption):

1. The client submits a job with secret: "" (empty) or omits the field entirely.
2. The server detects that KMS is enabled for this tenant.
3. A random 256-bit AES key (DEK) and a random 12-byte IV are generated.
4. The optimization result is serialized, compressed with bzip2, and encrypted with AES-256-GCM using the DEK and IV.
5. The encrypted bytes are stored in GridFS.
6. The DEK is wrapped (encrypted) by the tenant's KEK in the external KMS.
7. The wrapped DEK, the KEK identifier, the IV, and encMode: "KMS" are stored in the GridFS metadata sec block.
8. The plaintext DEK is discarded from server memory. Only the wrapped DEK is persisted.

During retrieval (decryption):

1. The client requests the result with no secret — just jobId and tenantId.
2. The server reads the sec block from the GridFS metadata and sees encMode: "KMS".
3. The wrapped DEK is sent to the KMS for unwrapping using the KEK identified by kekId.
4. The KMS returns the plaintext DEK.
5. The ciphertext is decrypted with AES-256-GCM using the unwrapped DEK and the stored IV.
6. On success, the decrypted bytes are decompressed and returned as the RestOptimization response.
7. The plaintext DEK is discarded from memory again.

KMS mode algorithm details

No PBKDF2 is involved in KMS mode — the DEK is already a proper 256-bit cryptographic key, not a human passphrase, so key derivation is unnecessary.

Why KMS mode?

• Zero client effort. The client doesn't need to manage, store, or remember any secrets. Encryption is invisible.
• Key rotation. The server operator can rotate the KEK in the KMS without re-encrypting existing data — old wrapped DEKs remain valid as long as the old KEK version is retained.
• Tenant offboarding. Deleting a tenant's KEK in the KMS makes all their data permanently unreadable — a clean, cryptographic guarantee.
• Audit trail. Cloud KMS services (Azure Key Vault, AWS KMS) provide detailed logs of every wrap/unwrap operation.
• HSM-backed security. In production, KEKs can be stored in FIPS 140-2 Level 2 (or higher) hardware security modules.

The tradeoff

With KMS mode, the server operator can decrypt the data (since the server has access to the KMS). If a client requires that even the server operator cannot read their data, they should use CLIENT mode and manage their own secret.

Server configuration

KMS mode is configured server-side. The client does not need to know or change anything.

For local development and testing, the local provider generates RSA-2048 key pairs in memory per tenant. Keys are lost on server restart — data encrypted during one session cannot be decrypted after a restart. This is intentional for testing purposes.

For production (Azure Key Vault), the azure provider uses DefaultAzureCredential and the CryptographyClient from the Azure SDK. Each tenant is mapped to a KEK stored in Azure Key Vault. The required Azure RBAC role on the keys is Key Vault Crypto User (wrap/unwrap only — no key deletion or creation).

Encryption Mode Summary

The following table summarizes how the three modes differ from the client's perspective:

In all three modes, metadata is always stored unencrypted and stream data (progress, status, warnings, errors) is always stored as unencrypted plain text.

What Gets Stored in MongoDB

Each job-mode optimization produces the following documents in MongoDB, all tagged with jobId and tenantId:

GridFS (result snapshot)

The full optimization result (compressed, optionally encrypted) is stored in GridFS. The metadata contains all the fields needed for querying, status checking, and decryption.

Unencrypted example (no secret, no KMS):

{
  "metadata": {
    "_contentType": "application/x-bzip2",
    "creator": "PUBLIC_CREATOR",
    "createdTimeStamp": 1774210149597,
    "ident": "JOpt-Run-1774127074120",
    "type": "OptimizationConfig<JSONConfig>",
    "expireAt": "2026-03-24T20:09:09.626Z",
    "status": {
      "statusDescription": "SUCCESS_WITH_SOLUTION",
      "error": "NO_ERROR",
      "status": "SUCCESS_WITH_SOLUTION"
    },
    "jobId": "696c31c3-f419-4918-8a65-faf1f769d460",
    "tenantId": "123456",
    "compression": "bzip2",
    "encrypted": false
  }
}

CLIENT mode example (non-empty secret provided):

{
  "metadata": {
    "_contentType": "application/octet-stream",
    "creator": "PUBLIC_CREATOR",
    "createdTimeStamp": 1774131229940,
    "ident": "JOpt-Run-1774127074120",
    "type": "OptimizationConfig<JSONConfig>",
    "expireAt": "2026-03-23T22:13:50.593Z",
    "status": {
      "statusDescription": "SUCCESS_WITH_SOLUTION",
      "error": "NO_ERROR",
      "status": "SUCCESS_WITH_SOLUTION"
    },
    "sec": {
      "encMode": "CLIENT",
      "iv": "dPrQge5LIDdPxEeg",
      "salt": "UVSGQfW40PybJ2HecBhjmg==",
      "encAlgo": "AES/GCM/NoPadding",
      "secretKeyFacAlgo": "PBKDF2WithHmacSHA256",
      "secretKeySpecAlgo": "AES",
      "iterationCount": 310000,
      "keyLength": 256
    },
    "jobId": "648d2724-3a77-47f4-b937-d3ab6abf2341",
    "tenantId": "123456",
    "compression": "bzip2",
    "encrypted": true
  }
}

KMS mode example (empty secret, KMS enabled on server):

{
  "metadata": {
    "_contentType": "application/octet-stream",
    "creator": "PUBLIC_CREATOR",
    "createdTimeStamp": 1774309926039,
    "ident": "KMS-Test-Run",
    "type": "OptimizationConfig<JSONConfig>",
    "expireAt": "2026-03-25T23:52:06.831Z",
    "status": {
      "statusDescription": "SUCCESS_WITH_SOLUTION",
      "error": "NO_ERROR",
      "status": "SUCCESS_WITH_SOLUTION"
    },
    "sec": {
      "encMode": "KMS",
      "iv": "lJYEnGQoDJnG1TEE",
      "encAlgo": "AES/GCM/NoPadding",
      "keyLength": 256,
      "salt": "",
      "secretKeyFacAlgo": "",
      "secretKeySpecAlgo": "AES",
      "iterationCount": 0,
      "wrappedDek": "u1ndsoDeGw3P494Jn3XU405Mr+g9EkBy...8A==",
      "kekId": "local-kms://keys/tenant-abc-123"
    },
    "jobId": "a299d547-7db9-40aa-856e-e338c9d08593",
    "tenantId": "tenant-abc-123",
    "compression": "bzip2",
    "encrypted": true
  }
}

Notice the key differences across the three modes: _contentType distinguishes compressed from encrypted data. The sec block is absent for unencrypted data, contains PBKDF2 fields for CLIENT mode, and contains wrappedDek + kekId for KMS mode. The encrypted flag provides a quick boolean check without parsing the sec block.

Stream collections (progress, status, warning, error)

Each stream document also carries jobId and tenantId, enabling the job-based read endpoints to filter directly. Stream persistence is controlled by the streamPersistenceStratgySetting in the persistenceSetting — if saveProgress is false, no progress documents are written.

Persistence Settings

The persistenceSetting in the request body controls what gets saved, whether encryption is active, and for how long data is retained. The configuration is identical to classic FAF:

"persistenceSetting": {
  "mongoSettings": {
    "enablePersistence": true,
    "secret": "",
    "expiry": "PT48H",
    "optimizationPersistenceStratgySetting": {
      "saveOnlyResult": false,
      "saveConnections": false
    },
    "streamPersistenceStratgySetting": {
      "saveProgress": true,
      "cycleProgress": true,
      "saveStatus": true,
      "cycleStatus": true,
      "saveWarning": true,
      "saveError": true
    }
  }
}

The secret field determines the encryption mode: set it to a strong passphrase for CLIENT mode, or leave it as "" to let the server use KMS (if enabled) or store without encryption. A non-empty client secret always takes priority over KMS.

For a detailed explanation of each field, see the Understanding the OptimizationPersistenceSetting object section in the classic FAF tutorial.

Error Handling

When an optimization fails asynchronously (after the 202 was already returned), the error is persisted to the database. The client discovers it by polling findStatus (which will show ERROR) or findError (which contains the error message). The findOptimization endpoint will return the error result snapshot with the optimizationStatus set to ERROR and the error message in the error field.

Agreement

For reading our license agreement and for further information about license plans, please visit www.dna-evolutions.com.

Authors

A product by dna-evolutions ©