Local LLM Deployment: gpt4all and Ollama

• The document describes two methods for local deployment of language models: gpt4all and Ollama + AnythingLLM.
• The first method (gpt4all) suits beginner users.
• The second (Ollama + AnythingLLM) caters to advanced users needing local knowledge bases.

gpt4all Deployment

• gpt4all eliminates the need for command-line operations.
• It supports various lightweight models.
• It accommodates basic reasoning tasks.
• Steps include installing gpt4all, selecting the appropriate system version (Windows/macOS/Linux), and following on-screen prompts.
• The website to download gpt4all is: https://gpt4all.io

DeepSeek Model Download in gpt4all

• Access the model search function within gpt4all.

Starting a Conversation with gpt4all

• After a model is downloaded, conversations can be initiated.
• The interface is on the left-hand side.
• Select a model from the available list.
• Conduct conversations with the AI to confirm successful setup.

Ollama + AnythingLLM Local Knowledge Base Deployment.

• Ollama deployment suits more advanced users.
• The Ollama website: https://ollama.ai
• Ollama can be combined with the Chrome plugin Page Assist for graphical interface conversations.
• Page Assist link: https://chromewebstore.google.com/detail/page-assist-%E6%9C%AC%E5%9C%B0-ai-%E6%A8%A1%E5%9E%8B%E7%9A%84-web/jfgfiigpkhlkbnfnbobbkinehhfdhndo.
• Ollama can also be paired with the local knowledge base tool AnythingLLM: https://AnythingLLM.com/.
• This combination provides powerful features, knowledge base support, and extensibility.

Ollama Installation Steps

• Installation steps here are demonstrated using a Mac system.

DeepSeek Model Download & Embedding in Ollama

• In Ollama, select "Models" or use the search bar to find DeepSeek-R1.
• Choose an appropriate model, such as the 1.5b version.
• Copy the command, paste it into the command line, and press enter.
• A "success" message indicates the model has downloaded.
• To download the embedding model, search for "nomic-embed-text".
• Then copy the command to the terminal and paste.

Changing the Model Location in Ollama

• The default is that Ollama installs models on the C drive.
• First, enter system settings: "Advanced System Settings", and then "Environment Variables".
• To change the location where the models are located, click "New.".
• In "Variable name" type ollama_MODELS.
• In the "Value" enter the new directory.
• Move files from C:\Users\XX.Ollama\models (where XX is your username) to whatever drive you chose.

AnythingLLM Deployment Steps

• Enter the AnythingLLM homepage and click "Download for desktop".
• Then select the appropriate system and click download.

Configuring AnythingLLM

• Select Ollama in the list and then select a model.
• Data processing and user research prompts can simply be skipped.
• Modify the interface language by clicking the wrench icon in the lower left corner.
• Select "Settings" and change "Display Language" to "Chinese".

Data Upload

• Start with uploading your data to AnythingLLM.
• In settings, tap "AI provider" and select Embedder options.
• Select "Ollama", for "Model type" fill "nomic-embed-text:latest", and then save.
• Click the upload button next to the workspace, then click Upload file, then select the file, and then click Save.
• Test the model by clicking on New Thread, type something related to the topic, and see the output.