Skip to content

Use Kanaries Cloud for free as students and educators

PrivateGPT: Offline GPT-4 That is Secure and Private

Viktor Zinchenko
Name
Viktor Zinchenko

Published on

In the realm of artificial intelligence (AI) and natural language processing (NLP), privacy often surfaces as a fundamental concern, especially when dealing with sensitive data. PrivateGPT, a groundbreaking development in this sphere, addresses this issue head-on. Designed to run locally without an internet connection, it ensures total privacy by preventing data from leaving your execution environment.

What is PrivateGPT?

PrivateGPT is an innovative tool that marries the powerful language understanding capabilities of GPT-4 with stringent privacy measures. Leveraging the strength of LangChain, GPT4All, LlamaCpp, Chroma, and SentenceTransformers, PrivateGPT allows users to interact with GPT-4, entirely locally. Let's delve into the nitty-gritty of setting up PrivateGPT and how to use it efficiently.

Setting Up PrivateGPT: Step-by-Step

Setting up PrivateGPT primarily involves two steps: installing requirements and configuring the environment. To start, ensure you have Python 3.10 or later installed on your machine.

Step 1: Install dependencies

pip3 install -r requirements.txt

Step 2: Download and place the Language Learning Model (LLM) in your chosen directory. The default model is ggml-gpt4all-j-v1.3-groovy.bin. However, any GPT4All-J compatible model can be used.

Step 3: Rename example.env to .env and edit the environment variables:

  • MODEL_TYPE: Specify either LlamaCpp or GPT4All.
  • PERSIST_DIRECTORY: Set the folder for your vector store.
  • MODEL_PATH: Provide the path to your LLM.
  • MODEL_N_CTX: Determine the maximum token limit for the LLM model.
  • EMBEDDINGS_MODEL_NAME: Specify the SentenceTransformers embeddings model name.
  • TARGET_SOURCE_CHUNKS: Determine the number of chunks that will be used to answer a question.

With the environment set up, we can now proceed to ingest the data.

Ingesting Data with PrivateGPT

PrivateGPT supports various file types ranging from CSV, Word Documents, to HTML Files, and many more. Here's how you ingest your own data:

Step 1: Place your files into the source_documents directory

Step 2: Run the ingest.py script to process all data

python ingest.py

The script creates a local embeddings database in the db folder. It takes about 20-30 seconds per document, depending on the document size.

Interacting with PrivateGPT

Now, let's dive into how you can ask questions to your documents, locally, using PrivateGPT:

Step 1: Run the privateGPT.py script:

python privateGPT.py

Step 2: When prompted, input your query.

Within 20-30 seconds, depending on your machine's speed, PrivateGPT generates an answer using the GPT-4 model and provides the sources it used from your documents to create the response.

Unleashing the Power of PrivateGPT: The Underlying Mechanics

PrivateGPT's privacy-preserving local operation is underpinned by a blend of potent tools. ingest.py harnesses LangChain tools to parse documents, creating local embeddings using HuggingFaceEmbeddings (part of SentenceTransformers). The resulting embeddings are stored in a local vector database with Chroma vector store.

privateGPT.py employs a local LLM — GPT4All-J or LlamaCpp — to comprehend user queries and fabricate fitting responses. The local vector store is used to extract context for these responses, leveraging a similarity search to find the corresponding context from the ingested documents.

System Requirements and Troubleshooting

PrivateGPT's system prerequisites include Python 3.10 or later. During the pip install process, if a C++ compiler error surfaces, installation instructions for both Windows 10/11 and Mac running Intel are provided.

You can access PrivateGPT GitHub here (opens in a new tab).

Conclusion

PrivateGPT stands as a testament to the fusion of powerful AI language models like GPT-4 and stringent data privacy protocols. It offers a secure environment for users to interact with their documents, ensuring that no data gets shared externally. Whether you're an AI enthusiast or a privacy-focused user, PrivateGPT opens up new possibilities for privacy-preserving AI applications.