8G. To download from a specific branch, enter for example TheBloke/Wizard-Vicuna-30B. Finally, and unrelated to the GGML, I then made GPTQ 4bit quantisations. For the first time ever, this means GGML can now outperform AutoGPTQ and GPTQ-for-LLaMa inference (though it still loses to exllama) Note: if you test this, be aware that you should now use --threads 1 as it's no longer beneficial to use. Under Download custom model or LoRA, enter TheBloke/Wizard-Vicuna-13B-Uncensored-SuperHOT-8K-GPTQ. cpp. Model card Files Community. I worked with GPT4 to get it to run a local model, but I am not sure if it hallucinated all of that. Their rate of progress is incredible. INFO:Loaded the model in 104. Here are the ggml versions: The unfiltered vicuna-AlekseyKorshuk-7B-GPTQ-4bit-128g-GGML and the newer vicuna-7B-1. GPTQ dataset: The dataset used for quantisation. cpp/GGML CPU inference, which enables lower cost hosting vs the standard pytorch/transformers-based GPU hosting. wo, and feed_forward. In the top left, click the refresh icon next to. bitsandbytes: VRAM Usage. 1 results in slightly better accuracy. GPTQ model: anon8231489123/vicuna-13b-GPTQ-4bit-128g on huggingfaceoriginal model: lm-. In addition to defining low-level machine learning primitives (like a tensor. ago. cpp team on August 21, 2023, replaces the unsupported GGML format. GGML vs. Please note that these GGMLs are not compatible with llama. I don't have enough VRAM to run the GPTQ one, I just grabbed the. With Transformers and TRL, you can: Quantize an LLM with GPTQ with a 4-bit, 3-bit, or 2-bit precision. Pygmalion 7B SuperHOT 8K GPTQ. TheBloke/Wizard-Vicuna-13B-Uncensored-GPTQ. after prompt ingestion). safetensors: 4: 128: False: 3. OpenChatKit is an open-source large language model for creating chatbots, developed by Together. Supports transformers, GPTQ, AWQ, EXL2, llama. Click the Model tab. Click Download. Because of the different quantizations, you can't do an exact comparison on a given seed. Its upgraded tokenization code now fully accommodates special tokens, promising improved performance, especially for models utilizing new special tokens and custom. Wait until it says it's finished downloading. The gpu is waiting for more work while cpu is maxed out. Locked post. GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Is it faster for inferences than the GPTQ format? You can't compare them because they are for different purposes. GPTQ-for-LLaMa vs bitsandbytes. For more general-purpose projects that require complex data manipulation, GPTQ's flexibility and extensive capabilities. went with 12,12 and that was horrible. GGML vs. bat to activate env, then from that browse to the AutoGPTQ and run the command - it should work. Note that the GPTQ dataset is not the same as the dataset. But for me, using Oobabooga branch of GPTQ-for-LLaMA AutoGPTQ versus llama-cpp-python 0. GPTQ is an alternative method to quantize LLM (vs llama. in the download section. Please specify it manually using --model_type argument Press any key to continue . Supported GGML models: LLAMA (All versions including ggml, ggmf, ggjt, gpt4all). 13B is parameter count, meaning it was trained on 13 billion parameters. NF4 — Due to the massive size of Large Language Models (LLMs), quantization has become an essential technique to run them efficiently. This repository contains the code for the ICLR 2023 paper GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers. When you run this program you should see output from the trained llama. 4. Some GPTQ clients have had issues with models that use Act Order plus Group Size, but this is generally resolved now. i understand that GGML is a file format for saving model parameters in a single file, that its an old problematic format, and. What's especially cool about this release is that Wing Lian has prepared a Hugging Face space that provides access to the model using llama. cpp is another framework/library that does the more of the same but specialized in models that runs on CPU and quanitized and run much faster. raw: Google GSheet with comments enabled. 13B is parameter count, meaning it was trained on 13 billion parameters. I think the gpu version in gptq-for-llama is just not optimised. Here's some more info on the model, from their model card: Model Description. OpenLLaMA is an openly licensed reproduction of Meta's original LLaMA model. {"payload":{"allShortcutsEnabled":false,"fileTree":{"":{"items":[{"name":". Using a dataset more appropriate to the model's training can improve quantisation accuracy. H2OGPT's OASST1-512 30B GGML These files are GGML format model files for H2OGPT's OASST1-512 30B. cpp. llama. Llama 2 is an open-source large language model (LLM) developed by Meta AI and Microsoft. For the first time ever, this means GGML can now outperform AutoGPTQ and GPTQ-for-LLaMa inference (though it still loses to exllama) Note: if you test this, be aware that you should now use --threads 1 as it's no longer beneficial to use. 5-16K-GPTQ via AutoGPTQ which should theoretically give me same results as the same model of GGUF type but with even better speeds. 4bit and 5bit GGML models for CPU inference. 55 tokens/s Falcon, unquantised bf16: Eric's base WizardLM-Falcon: 27. Quantize your own LLMs using AutoGPTQ. New comments cannot be posted. 1-GPTQ-4bit-128g-GGML. GPU/GPTQ Usage. In practice, GPTQ is mainly used for 4-bit quantization. 1-GPTQ-4bit-128g-GGML. AWQ, on the other hand, is an activation-aware weight quantization approach that protects salient weights by. All 3 versions of ggml LLAMA. 1. 1 GPTQ 4bit runs well and fast, but some GGML models with 13B 4bit/5bit quantization are also good. Training Details. Looks like the zeros issue corresponds to a recent commit to GPTQ-for-LLaMa (with a very non-descriptive commit message) which changed the format. 8k • 427 TheBloke/OpenHermes-2. Train. So the end. This adds full GPU acceleration to llama. I have high hopes for an unfiltered mix like this, but until that's done, I'd rather use either vicuna-13b-free or WizardLM-7B-Uncensored alone. TheBloke/guanaco-65B-GGML. I haven't tested the memory. GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Supports transformers, GPTQ, AWQ, EXL2, llama. GPTQ and ggml-q4 both use 4-bit weights, but differ heavily in how they do it. According to open leaderboard on HF, Vicuna 7B 1. Enterprises using it as an alternative to GPT-4 if they can fine-tune it for a specific use case and get comparable performance. safetensors along with all of the . I have even tried the vicuna-13B-v1. 4375 bpw. For illustration, GPTQ can quantize the largest publicly-available mod-els, OPT-175B and BLOOM-176B, in approximately four GPU hours, with minimal increase in perplexity, known to be a very stringent accuracy metric. Then the new 5bit methods q5_0 and q5_1 are even better than that. If your cpu (the core that is running python inference) is at 100% and gpu is 25%, the bottleneck is cpu. jsons and . GPTQ dataset: The dataset used for quantisation. 0. Even with the latest version (0. TheBloke/SynthIA-7B-v2. These files are GGML format model files for Eric Hartford's Wizard Vicuna 13B Uncensored. py <path to OpenLLaMA directory>. 1 results in slightly better accuracy. text-generation-webui - A Gradio web UI for Large Language Models. cpp GGML models, so we can compare to figures people have been doing there for a. Damp %: A GPTQ parameter that affects how samples are processed for quantisation. Using MythoLogic-L2's robust understanding as its input and Huginn's extensive writing capability as its output seems to. For GPTQ I had to have a GPU, so I went back to that 2 x 4090 system @ $1. cpp / GGUF / GGML / GPTQ & other animals. GGML is a C library for machine learning (ML) — the “GG” refers to the initials of its originator (Georgi Gerganov). First, we explore and expand various areas in the same topic using the 7K conversations created by WizardLM. Llama 2 Airoboros 7/13/70B GPTQ/GGML Released! Find them on TheBloke's huggingface page! Hopefully, the L2-70b GGML is an 16k edition, with an Airoboros 2. Immutable fedora won't work, amdgpu-install need /opt access If not using fedora find your distribution's rocm/hip packages and ninja-build for gptq. I think that's a good baseline to. github","path":". q6_K version of the model (llama. Using MythoLogic-L2's robust understanding as its input and Huginn's extensive writing capability as its output seems to. As far as I'm aware, GPTQ 4-bit w/ Exllama is still the best option. My CPU is an "old" Threadripper 1950X. ggml is a tensor library for machine learning to enable large models and high performance on commodity hardware. Anyone know how to do this, or - even better - a way to LoRA train GGML directly?gptq_model-4bit-128g. if you have oobabooga one click install, run cmd_windows. Once it's finished it will say "Done". GPTQ is currently the SOTA one shot quantization method for LLMs. This video explains difference between GGML and GPTQ in AI models in very easy terms. Try 4bit 32G and you will more than likely be happy with the result! When comparing GPTQ-for-LLaMa and llama. Click the Model tab. Compare privateGPT vs GPTQ-for-LLaMa and see what are their differences. 60 GB: 6. They take only a few minutes to create, vs more than 10x longer for GPTQ, AWQ, or EXL2, so I did not expect them to appear in any Pareto frontier. Eventually, this gave birth to the GGML format. That being said, given that ggml is now outdated and gguf is the new version I don’t know if that is still the case. Testing the new BnB 4-bit or "qlora" vs GPTQ Cuda upvotes. Recent advancements in weight quantization allow us to run massive large language models on consumer hardware, like a LLaMA-30B model on an RTX 3090 GPU. To use with your GPU using GPTQ pick one of the . Before you can download the model weights and tokenizer you have to read and agree to the License Agreement and submit your request by giving your email address. Quantization-Aware Training (QAT) A technique that refines the PTQ model to maintain accuracy even after quantization. I appreciate that alpaca models aren't generative in intent, and so perplexity is not a good measure. GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. 1]}. vw and feed_forward. GPTQ dataset: The dataset used for quantisation. For GPTQ tests, I used models with groupsize 128 and no desc_act, which are the ones that are widely used. I'm also still a bit curious of GGML is competitive with GPTQ/exllama when running on Nvidia GPU. GGUF / GGML versions run on most computers, mostly thanks to quantization. GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. GGML vs. Text Generation • Updated Sep 27 • 15. GGML - Large Language Models for Everyone: a description of the GGML format provided by the maintainers of the llm Rust crate, which provides Rust bindings for GGML. CPU is generally always 100% on at least one core for gptq inference. Click Download. However, llama. model files. GPTQ vs. 58 seconds. 9 min read. GGML vs. 4bit and 5bit GGML models for GPU inference. We will try to get in discussions to get the model included in the GPT4All. 2k 3. Gptq-triton runs faster. Bitsandbytes can perform integer quantization but also supports many other formats. Looks like the zeros issue corresponds to a recent commit to GPTQ-for-LLaMa (with a very non-descriptive commit message) which changed the format. . 0-GPTQ. The change is not actually specific to Alpaca, but the alpaca-native-GPTQ weights published online were apparently produced with a later version of GPTQ-for-LLaMa. i did the test using theblokes 'TheBloke_guanaco-33B-GGML' vs 'TheBloke_guanaco-33B-GPTQ'. 5 (16k) is fine-tuned from Llama 2 with supervised instruction fine-tuning and linear RoPE scaling. so thank you so much for taking the time to post this. cpp GGML models, so we can compare to figures people have been doing there for a while. q4_0. Edit model. It's a single self contained distributable from Concedo, that builds off llama. Block scales and mins are quantized with 4 bits. Maybe now we can do a vs perplexity test to confirm. The model will start downloading. Scales and mins are quantized with 6 bits. Update to include TheBloke_Wizard-Vicuna-13B-Uncensored-GPTQ GPTQ-for-LLaMa VS Auto GPTQ VS ExLlama (This does not change GGML test results. I'm also still a bit curious of GGML is competitive with GPTQ/exllama when running on Nvidia GPU. As this is a GPTQ model, fill in the GPTQ parameters on the right: Bits = 4, Groupsize = 128, model_type = Llama. pt: Output generated in 113. One option to download the model weights and tokenizer of Llama 2 is the Meta AI website. 33B you can only fit on 24GB VRAM, even 16Gb are not enough. Llama 2. I’ve tried the 32g and 128g and both are problematic. NF4. Enjoy using the L2-70b variants but don't enjoy the occasional 8 minute wait of a full cublas context refresh lol. One of the most popular is GPTQ – introduced in March 2023 which uses 4 bits (16 distinct values!) to represent a floating point. This might help get a 33B model to load on your setup but you can expect shuffling between VRAM and system RAM. Open the text-generation-webui UI as normal. the latest version should be 0x67676d66, the old version which needs migration should be: 0x67676d6c. . marella/ctransformers: Python bindings for GGML models. This end up using 3. GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. The GGML_TYPE_Q5_K is a type-1 5-bit quantization, while the GGML_TYPE_Q2_K is a type-1 2-bit quantization. from_pretrained ("TheBloke/Llama-2-7b-Chat-GPTQ", torch_dtype=torch. 1-AWQ for. Llama-2-7B-32K-Instruct is an open-source, long-context chat model finetuned from Llama-2-7B-32K, over high-quality instruction and chat data. text-generation-webui - A Gradio web UI for Large Language Models. Deploy. For example, from here: TheBloke/Llama-2-7B-Chat-GGML TheBloke/Llama-2-7B-GGML. 256 70 2,931 contributions in the last year Contribution Graph; Day of Week: November Nov: December Dec: January Jan: February Feb: March Mar: April Apr: May May: June Jun:. Renamed to KoboldCpp. GPTQ is a one-shot weight quantization method based on approximate second-order information, allowing for highly accurate and efficient quantization of GPT models with 175 billion parameters. . GPTQ. TheBloke/SynthIA-7B-v2. ) Test 3 TheBloke_Wizard-Vicuna-13B-Uncensored-GPTQ GPTQ-for-LLaMa The first one is to be installed when you want to load and interact with GPTQ models; the second one is to be ued with GGUF/GGML files, that can run on CPU only. Quantized in 8 bit requires 20 GB, 4 bit 10 GB. pt. Step 2. GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. In addition to defining low-level machine learning primitives (like a tensor. I have an Alienware R15 32G DDR5, i9, RTX4090. We will provide a comprehensive guide on how to implement GPTQ using the AutoGPTQ library. As for when - I estimate 5/6 for 13B and 5/12 for 30B. Benchmark Execution: Running benchmarks on identical tasks using both SYCL and CUDA forms the foundation of performance comparison. This user has. It loads in maybe 60 seconds. Step 1. The huge thing about it is that it can offload a selectable number of layers to the GPU, so you can use whatever VRAM you have, no matter the model size. GGML 30B model VS GPTQ 30B model 7900xtx FULL VRAM Scenario 2. If everything is configured correctly, you should be able to train the model in a little more than one hour (it. 8% pass@1 on HumanEval. Using a dataset more appropriate to the model's training can improve quantisation accuracy. Stars - the number of stars that a project has. GPTQ is better, when you can fit your whole model into memory. This will produce ggml-base. 30 43,757 7. GGML/GGUF models are tailored to minimize memory usage rather than prioritize speed. Unique Merging Technique. There's just something unusual/different causing it not to work for you guys as a GPTQ on Windows. So here it is, after exllama, GPTQ and SuperHOT stole GGML the show for a while, finally there's a new koboldcpp version with: full support for GPU acceleration using CUDA and OpenCL. Teams. Repositories available 4bit GPTQ models for GPU inference. Do you know of any github projects that I could replace GPT4All with that uses CPU-based GPTQ in Python? TheBloke/guanaco-65B-GPTQ. But GGML allows to run them on a medium gaming PC at a speed that is good enough for chatting. Disclaimer: The project is coming along, but it's still a work in progress! Hardware requirements. ローカルLLMの量子化フォーマットとしては、llama. It completely replaced Vicuna for me (which was my go-to since its release), and I prefer it over the Wizard-Vicuna mix (at least until there's an uncensored mix). SuperHOT is a new system that employs RoPE to expand context beyond what was originally possible for a model. が、たまに量子化されてい. GitHub Copilot's extension generates a multitude of requests as you type, which can pose challenges, given that language models typically process one. The benchmark was run on a NVIDIA-A100 instance and the model used was TheBloke/Mistral-7B-v0. Model Developers Meta. cpp (GGUF), Llama models. Model Architecture Llama 2 is an auto-regressive language model that uses an optimized transformer architecture. ago. Untick Autoload the model. github. GGCC is a new format created in a new fork of llama. (2) Es ist schwer zu sagen wann man lieber auf ein GPTQ quantisierten oder einen. text-generation-webui - A Gradio web UI for Large Language Models. Under Download custom model or LoRA, enter TheBloke/Nous-Hermes-13B-GPTQ. This is wizard-vicuna-13b trained with a subset of the dataset - responses that contained alignment / moralizing were removed. I've used these with koboldcpp, but CPU-based inference is too slow for regular usage on my laptop. nf4 without double quantization significantly uses more memory than GPTQ. 0. I've been trying to try different ones, and the speed of GPTQ models are pretty good since they're loaded on GPU, however I'm not sure which one would be the best option for what purpose. ) Prompts Various (I'm not actually posting the question/answers it's irreverent for this test as we are checking speeds. Click Download. By reducing the precision ofGGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. So I need to train a non-GGML, then convert the output. This is probably stupid and maybe ggml already works this way, but I am wondering, since the main bottleneck seems to be memory bandwidth, could the batches be processed in. I appear to be stuck. Damp %: A GPTQ parameter that affects how samples are processed for quantisation. Pygmalion 7B SuperHOT 8K GGML. This is self. KoboldAI (Occam's) + TavernUI/SillyTavernUI is pretty good IMO. TheBloke/MythoMax-L2-13B-GPTQ VS Other Language Models. In the top left, click the refresh icon next to Model. ggmlv3. Note that the 4-element list of dimensions uses 1 as a placeholder for unused dimensions - this is because the product of the dimensions should not equal zero. marella/ctransformers: Python bindings for GGML models. gpt4-x-vicuna-13B-GGML is not uncensored, but. 9. cpp. 兼容性最好的是 text-generation-webui，支持 8bit/4bit 量化加载、GPTQ 模型加载、GGML 模型加载、Lora 权重合并、OpenAI 兼容API、Embeddings模型加载等功能，推荐！. AWQ vs. So it seems that GPTQ has a similar latency problem. Bitsandbytes can perform integer quantization but also supports many other formats. cpp (GGUF), Llama models. I found its behavior extremely weird - whenever I use this to offload to my 12GB VRAM buffer - regardless of model size, the loader keeps pegging my RAM budget until Windows has had enough. cpp's GGML) that has awesome performance but supports only GPU acceleration. from_pretrained ("TheBloke/Llama-2-7b-Chat-GPTQ", torch_dtype=torch. even took the time to try all the versions of the ggml bins. It is a replacement for GGML, which is no longer supported by llama. smspillaz/ggml-gobject: GObject-introspectable wrapper for use of GGML on the GNOME platform. 0更新【6. My machine has 8 cores and 16 threads so I'll be. Using a dataset more appropriate to the model's training can improve quantisation accuracy. bin file is to use this script and this script is keeping the GPTQ quantization, it's not converting it into a q4_1 quantization. ) There's no way to use GPTQ on macOS at this time. 19】:1. GGUF) Thus far, we have explored sharding and quantization techniques. GPTQ scores well and used to be better than q4_0 GGML, but recently the llama. safetensors along with all of the . A simple one-file way to run various GGML and GGUF models with KoboldAI's UI llama. GPTQ vs. However, on 8Gb you can only fit 7B models, and those are just dumb in comparison to 33B. < llama-30b FP16 2nd load INFO:Loaded the model in 39. To use with your GPU using GPTQ pick one of the . The original WizardLM, a 7B model, was trained on a dataset of what the creators call evolved instructions. Learning Resources:TheBloke Quantized Models - from Hugging Face (Optimum) -. cpp you can also consider the following projects: gpt4all - gpt4all: open-source LLM chatbots that you can run anywhere. Due to the massive size of Large Language Models (LLMs), quantization has become an essential technique to run them efficiently. Pygmalion 13B SuperHOT 8K GPTQ. 5 if they can get it to be cheaper overall. 9 GB: True: AutoGPTQ: Most compatible. KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models. About GGML. cpp (GGUF/GGML)とGPTQの2種類が広く使われている。. 01 is default, but 0. H2OGPT's OASST1-512 30B GGML These files are GGML format model files for H2OGPT's OASST1-512 30B. pt file into a ggml. In this case, you might try something like the following: llama2-base-13b-kimono. Click Download. 2) and a Wikipedia dataset. gpt4-x-alpaca’s HuggingFace page states that it is based on the Alpaca 13B model, fine. 1 results in slightly better accuracy. Now, I've expanded it to support more models and formats. GPTQ means it will run on your graphics card at 4bit (vs GGML which runs on CPU, or the non-GPTQ version which runs at 8bit). This is an example to launch koboldcpp in streaming mode, load a 8k SuperHOT variant of a 4 bit quantized ggml model and split it between the GPU and CPU. It's true that GGML is slower. Supports NVidia CUDA GPU acceleration. ExLlamaV2 is a library designed to squeeze even more performance out of GPTQ. alpaca-lora - Instruct-tune LLaMA on consumer hardware. ggml's distinguishing feature is efficient operation on CPU. The older GGML format revisions are unsupported and probably wouldn't work with anything other than KoboldCCP since the Devs put some effort to offer backwards compatibility, and contemporary legacy versions of llamaCPP. 0. cpp. cpp team have done a ton of work on 4bit quantisation and their new methods q4_2 and q4_3 now beat 4bit GPTQ in this benchmark. Uses that GPT doesn’t allow but are legal (for example, NSFW content) Enterprises using it as an alternative to GPT-3. Prompt processing speed. 3 Python text-generation-webui VS llama Inference code for LLaMA modelsIt still works with Pygmalion 7B GPTQ, but it doesn't seem to work with Wizard Vicuna 13B GGML, although I can load and use the latter in Ooba. Try 4bit 32G and you will more than likely be happy with the result!GGML vs. 0. Reply reply. However, on 8Gb you can only fit 7B models, and those are just dumb in comparison to 33B. Another advantage is the. GPTQ has been very popular to create models in 4-bit precision that can efficiently run on GPUs. The lower bit quantization can reduce the file size and memory bandwidth requirements, but also introduce more errors and noise that can affect the accuracy of the model. Note: Download takes a while due to the size, which is 6. Click the Refresh icon next to Model in the top left. Nevertheless, there is no impediment to running GGUF on a GPU; in fact, it runs even faster compared to CPU execution. GGML is the only option on Mac. 9 min read. 4375 bpw. 0 dataset. Pick yer size and type! Merged fp16 HF models are also available for 7B, 13B and 65B (33B Tim did himself. Download 3B ggml model here llama-2–13b-chat. Currently I am unable to get GGML to work with my Geforce 3090 GPU. This script duplicates the addend and scale to match ggml's expectations, at the cost of wasting some memory. Links to other models can be found in the index at the bottom. For Kobold CCP you use GGML files insted of the normal gptq or f16 formats. This was to be expected. Learn more about TeamsRunning a 3090 and 2700x, I tried the GPTQ-4bit-32g-actorder_True version of a model (Exllama) and the ggmlv3. Supports transformers, GPTQ, AWQ, EXL2, llama. Looking forward, our next article will explore the GPTQ weight quantization technique in depth. OpenLLaMA is an openly licensed reproduction of Meta's original LLaMA model. To recap, every Spark. 0. Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits. GPTQ dataset: The dataset used for quantisation. Click the Model tab. Tim Dettmers' Guanaco 33B GGML These files are GGML format model files for Tim Dettmers' Guanaco 33B. Along with most 13B models ran in 4bit with around Pre-layers set to 40 in Oobabooga. 4bit quantization – GPTQ / GGML. It allowed models to be shared in a single file, making it convenient for users. yaml. ML Blog - 4-bit LLM Quantization with GPTQI think it's still useful - GPTQ or straight 8-bit quantization in Transformers are tried and tested, and new methods might be buggier. Finding a way to try GPTQ to compareIt is strongly recommended to use the text-generation-webui one-click-installers unless you're sure you know how to make a manual install.