Blog | Developer | | 13 min read

Is Your Offline AI Stack Truly Offline?

Is Your Offline AI Stack Truly Offline?

Key Takeaways

  • On-premises AI is not automatically air-gapped: a self-hosted RAG stack still attempted 52 outbound connections with cached models and 32 during cold-start testing.
  • The outbound calls came from Ollama’s model registry, Hugging Face model checks and downloads, Arize Phoenix’s documentation service, and a VectorAI DB deployment-usage notification.
  • Cold-start testing failed because the stack could not download model files under a deny-all firewall, so models must be staged locally before isolation.
  • Offline settings (OLLAMA_NO_CLOUD, HF_HUB_OFFLINE, and Phoenix telemetry and external-resource flags) cut blocked packets from 52 to one without affecting query answers.
  • An egress audit with DNS logs, packet captures, and firewall logs reveals hidden dependencies, and the team’s reproducible repository lets you run it against your own stack.

On-premises AI is not the same as air-gapped AI. A truly air-gapped AI deployment has zero external network path, with all the dependencies needed for operation available locally. Running a Large Language Model (LLM) and vector database on your own infrastructure does not automatically provide that isolation.

An on-premises deployment can still contact external model registries, check for software updates, send telemetry to a public endpoint, or route prompts to hosted APIs. To establish whether your stack attempts external communication, you need to inspect its network activity.

We blocked external connections on a representative local Retrieval-Augmented Generation (RAG) stack by using a deny-by-default firewall. The stack still attempted outbound connections, even though it could answer queries offline. We captured those attempts, identified their destinations, and packaged a reproducible egress audit you can run against your own self-hosted AI stack to find hidden network dependencies.

TL;DR

  • Self-hosted AI does not guarantee zero egress. Our audit captured 52 blocked packets in a self-hosted RAG application with cached models and 32 without them.
  • Pre-staging models helps reduce external dependencies. In our experiment, the RAG application answered queries with cached models but failed during cold-start testing because it couldn’t download the required files.
  • Enforce offline settings and verify them with an egress audit. Our hardened configuration reduced blocked packets from 52 to one without affecting query execution.

What We Saw When We Cut the Cord

In our pre-staged run, we traced outbound attempts to three external hostnames across Ollama, the RAG application, and Arize Phoenix. During cold-start testing, we recorded requests to five hostnames, including VectorAI DB’s deployment-usage endpoint.

We tested a local RAG stack built with LlamaIndex, Actian VectorAI DB Community Edition for retrieval, Ollama running Llama 3.2 3B, a local Hugging Face embedding model, and Arize Phoenix for tracing. All four application components ran in Docker containers on Ubuntu 24.04 under WSL2.

We used a deny-by-default firewall to block external IPv4 traffic while preserving communication between containers. A Domain Name System (DNS) sinkhole recorded requested hostnames, while tcpdump and firewall logs captured connection attempts and blocked packets. We ran the same patient-records query in a pre-staged condition with model files cached locally, and a cold-start condition with the caches removed. You can reproduce the setup using the GitHub repository. The table below shows the outbound dependencies we identified across the RAG components during both restricted runs.

Component Call type Requested destination  Pre-staged packets Cold-start packets 
Ollama Cloud catalog and model registry ollama.com, registry.ollama.ai 3 4
RAG app / Hugging Face libraries  Model metadata checks and downloads huggingface.co 48 26
Arize Phoenix  Remote documentation service arizeai-433a7140.mintlify.app 1 1
Actian VectorAI DB  Deployment usage notification elspv.pervasive.com 0 1
Total 52 32

Counts reflect blocked packets and retries, rather than unique requests. During cold-start testing, the packet capture recorded 31 outbound attempts, and the firewall log recorded an additional blocked packet from VectorAI DB.

With the model files already cached, the pre-staged run recorded 68 DNS lookups and 52 blocked packets. The RAG application recorded 48 blocked packets associated with Hugging Face, followed by Ollama with three packets and Arize Phoenix with one packet. VectorAI DB made no observed outbound attempts during this run. Despite the blocked connections, the RAG application retrieved the relevant patient-records policy and successfully answered the query, as shown in the reproduced test in Figure 2.

pre-staged egress audit

Pre-staged egress audit showing 68 DNS lookups and 52 blocked packets across Ollama, the RAG application, and Arize Phoenix

reproduced pre-staged rag test

Reproduced pre-staged RAG test showing the application loading local documents, indexing them into VectorAI DB, and successfully answering a patient-records query with external IPv4 egress blocked

With the model caches removed in the cold-start run, Ollama attempted to reach its registry and the embedding libraries tried to download model files from Hugging Face. Both requests failed under the deny-all firewall, preventing the RAG application from completing the query. The firewall recorded 32 blocked packets and 61 DNS lookups across the stack. VectorAI DB also attempted one connection to elspv.pervasive.com, a deployment-usage notification that does not affect local storage or retrieval.

cold start egress audit

Cold-start egress audit showing 61 DNS lookups and 32 blocked packets

Which Outbound Calls Matter Most?

Calls that transmit prompts or document chunks to external services pose the highest confidentiality risk. Our egress firewall blocked the observed outbound attempts, but those requests differed in purpose and potential impact. A model registry check may reveal information about the software you are running, whereas a hosted embedding API could receive sensitive document chunks.

The table below breaks down when each outbound call occurs and the risks it could introduce to an offline AI deployment.

Outbound activity  Observed in our audit? When it occurs  Primary risk
Ollama model registry pull Yes, cold-start Setup-time Requires external access to download model files that have not been staged locally
Ollama external service request Yes Runtime, startup May disclose information about the requesting deployment and violates zero-egress requirements
Hugging Face model download  Yes, cold-start Setup-time Missing local files can prevent the RAG application from starting successfully
Hugging Face metadata checks  Yes, pre-staged Runtime, during model loading May reveal model identifiers and usage metadata
Arize Phoenix documentation request  Yes Runtime, at startup Contacts an external service, but no trace-data exposure established
VectorAI DB deployment-usage notification Yes, cold-start Periodic background activity Nonessential to local retrieval, but violates strict zero-egress requirements when attempted
Hosted embedding or inference API  No, hypothetical Runtime, if configured Can transmit prompts or sensitive document content to an external provider

Ollama and Hugging Face outbound behavior illustrate why you need to separate setup dependencies from runtime checks. During our cold-start run, Ollama could not retrieve its model manifest, and the RAG application could not download its embedding model. In the pre-staged run, both models were available locally, yet their supporting libraries continued attempting external catalog and metadata requests. According to the Hugging Face Hub documentation⁠, hf_hub_download can still check for newer versions of files already stored in the local cache.

For your deployment, check whether model-loading libraries still contact public registries when the weights are cached locally. Review startup requests and retries, as they can trigger outbound connections when you have staged the required models.

Background services introduce another category of outbound activity. Arize Phoenix attempted to contact its hosted documentation service during startup, and VectorAI DB initiated a periodic deployment-usage notification. VectorAI DB’s request did not affect offline vector storage or retrieval, which continued working with external egress blocked. But you should identify similar outbound attempts when auditing your own stack, because nonessential background connections still violate zero-egress requirements.

Another potential outbound concern is the use of hosted embedding APIs. For example, LlamaIndex Python falls back to OpenAI’s hosted embedding integration by default when you do not point Settings.embed_model to a local embedding model. If your AI pipeline sends confidential document chunks to that API, you are transmitting sensitive information outside your infrastructure. We configured a local embedding model for our experiment, and our firewall blocked the outbound connection attempts we captured.

Before running your own stack with sensitive data, check which providers handle embeddings and inference, inspect the data your application sends to them, and confirm whether any component can fall back to a hosted API. Treat any runtime call that could send prompts, document chunks, or other sensitive workload data to an unapproved external service as a potential breach. Block those connections at the network boundary and document any remaining outbound attempts.

What it Takes to Close the Gap

An air-gapped AI enclave requires network isolation, locally available dependencies, and controls that prevent components from reaching external services. Start with a deny-by-default egress firewall, remove external network routes, and restrict DNS resolution to approved internal services. These controls establish the network boundary, while application-level settings address the outbound dependencies your audit identifies.

For open-weight models, download and stage the required files before isolating the environment. In our hardened rerun, we set OLLAMA_NO_CLOUD=1 to disable Ollama’s cloud features and HF_HUB_OFFLINE=1 to prevent Hugging Face libraries from making HTTP requests to the Hub. With the model files already cached, both components could use local resources without repeating the registry checks recorded in our earlier tests.

Observability tools process prompts, retrieved context, and application traces, so you need to review their external connections when isolating your AI stack. We configured Arize Phoenix with PHOENIX_ALLOW_EXTERNAL_RESOURCES=false and PHOENIX_TELEMETRY_ENABLED=false to restrict external resources and disable telemetry. Apply the same scrutiny to any monitoring or tracing service in your deployment, especially when it touches sensitive workloads.

A production air-gapped environment also needs a controlled update process. Model weights, container images, packages, and security patches should enter the isolated environment through approved, verified, and signed physical media, with each transfer documented. This allows you to update the AI stack without connecting it to public registries.

Our experiment tested firewall-enforced isolation, DNS logging, and application-level offline settings. The additional network and update controls are recommendations for production deployments.

The diagram below is a reference architecture for an air-gapped RAG deployment, incorporating the controls tested in our experiment and those recommended for production environments.

air gapped rag enclave

An air-gapped RAG enclave with local inference and retrieval, deny-by-default network controls, and a signed-media update path

We applied these offline settings to the test stack and repeated the pre-staged audit under deny-all egress. During our 32-minute observation window, the hardened run recorded only one blocked connection from VectorAI DB’s deployment-usage notification, with no outbound attempts from Ollama, the RAG application, or Arize Phoenix. The configuration changes reduced the blocked packets from 52 to one, and DNS lookups from 68 to two, while the firewall stopped the remaining attempt from VectorAI DB. The app also answered the test query correctly.

hardened egress audit

Hardened egress audit showing the resulting log after applying offline settings to the RAG components

Step-by-Step Egress Audit

Run an egress audit against your on-premises AI stack to identify external connections and verify that your application continues working when you block outbound traffic.

Before you begin

You need a Linux environment with Docker Engine, Docker Compose, iptables, and tcpdump. We ran our experiment on Ubuntu 24.04 under WSL2.

To reproduce our setup, clone the repository and open the project directory:

git clone https://github.com/actian-devs/developer-content.git
cd developer-content/blog/egress-audit

Follow the README to stage the Ollama and Hugging Face models while internet access is available. Start the stack using the base Compose file before applying the audit overlay, which adds the DNS sinkhole. After caching the models and confirming that the RAG application can answer the sample query, proceed with the egress audit.

Step 1: Block outbound traffic and start monitoring

Keep your stack’s internal network running to allow communication between components without permitting external connections. For our Docker deployment on Ubuntu 24.04, we used a logging DNS sinkhole and added firewall rules to Docker’s DOCKER-USER chain. The sinkhole recorded requested hostnames, and the firewall logged and rejected outbound packets.

First, configure Docker Compose to use the base file and the audit overlay, which adds the DNS sinkhole:

export COMPOSE_FILE=docker-compose.yml:docker-compose.audit.yml

Run this command in both terminals you use for the audit. Then prepare the containers without starting them:

docker compose down
docker compose up --no-start

Activate the firewall and clear the previous DNS and kernel logs in your controlled test environment so the new records correspond to this run:

sudo ./audit/egress-deny.sh on
sudo truncate -s 0 capture/dns.log
sudo dmesg -C

The firewall policy permits communication between containers on br-rag but restricts external IPv4 egress. In your second terminal, start packet capture before launching the services.

sudo tcpdump -i br-rag -nn -s 0 -w capture/prestaged.pcap 2>&1 | tee -i capture/prestaged.tcpdump.txt

Return to the first terminal, start the stack, and execute the test query.

docker compose up -d
docker compose run --rm app --question "How long are adult patient records kept?"

Keep the capture running for another 30 minutes after the query completes to record scheduled retries and background requests.

sleep 1800

Stop tcpdump with Ctrl+C in the second terminal. You’ll find the DNS sinkhole configuration and complete firewall policy in the GitHub repository.

Step 2: Identify and classify every outbound caller

Use the DNS logs to identify requested hostnames, packet captures to inspect connection attempts, and firewall logs to confirm what was blocked. Give each container a fixed IP address so you can associate network traffic with the originating component.

For example, our Docker Compose configuration assigns Ollama this address:

ollama:
  networks:
    rag:
      ipv4_address: 10.77.0.11

After stopping the packet capture, save the DNS and firewall logs from the completed run. The GitHub repository includes the commands for collecting each log.

sudo cp capture/dns.log capture/prestaged.dns.log
sudo TZ=UTC dmesg --time-format iso > capture/prestaged.kernel.log

Then use the audit summarizer to group outbound activity by component and destination.

sudo ./audit/summarize.sh capture/prestaged

The summarizer reads the DNS log, packet capture, firewall log, and tcpdump statistics saved in Step 1. The GitHub repository contains the complete script.

Classify each attempt as a setup dependency, runtime check, or background request. Run the audit with your models pre-staged, then repeat with empty caches in a controlled test to identify dependencies that would prevent an offline deployment from starting.

Step 3: Disable unnecessary external dependencies

Use the recorded destinations and component logs to identify which outbound features you should disable. Then apply the corresponding offline settings in a Docker Compose override file.

For our RAG stack, we used:

services:
  ollama:
    environment:
      - OLLAMA_NO_CLOUD=1

  phoenix:
    environment:
      - PHOENIX_ALLOW_EXTERNAL_RESOURCES=false
      - PHOENIX_TELEMETRY_ENABLED=false

  app:
    environment:
      - HF_HUB_OFFLINE=1

Check your application code for hosted API defaults and explicitly configure local embeddings and inference. In our RAG application, we set Settings.embed_model to a local Hugging Face embedding model.

Stage the required model weights and embedding files locally before enforcing any offline settings. Your application must be able to load them without contacting an external registry. Some background requests may not have application-level settings to disable them. In those cases, retain the network restriction and document the remaining attempts.

Step 4: Re-test and preserve the audit evidence

Repeat the pre-staged query with your offline settings and deny-all firewall active. Then compare the results with your initial capture to see how the outbound activity has changed.

For a complete audit record, save the DNS logs, packet captures, firewall logs, and successful query output alongside the configuration used for each run. These records show your auditor that the application worked without external connections and provide a trace of any remaining outbound attempts.

The egress audit repository includes the Docker Compose files, cache-staging procedure, firewall policy, capture commands, and results from all three test conditions.

Wrapping Up

Before you call your AI stack offline, test what happens when you remove its network access. You might find model registries, background checks, or external services that your local deployment still depends on.

In our RAG experiment, VectorAI DB handled the retrieval layer locally, even with external connections blocked. The application could retrieve relevant document chunks without relying on a hosted vector database.

If you’re building an offline RAG system, sign up for VectorAI DB Community Edition and run the egress audit against your own workload. The audit will reveal external network dependencies you need to resolve before deploying your AI stack in an air-gapped environment.

Join Discord