| license: apache-2.0 | |
| language: | |
| - en | |
| base_model: | |
| - meta-llama/Llama-3.1-8B-instruct | |
| pipeline_tag: text-generation | |
| tags: | |
| - lora | |
| - adapter | |
| - writing | |
| - CoT | |
| - unsloth | |
| ## Model Details | |
| - Base Model: meta-llama/Llama-3.1-8B-instruct | |
| ## Merger Configuration | |
| ### Source Adapters | |
| All source adapters share the following configuration: | |
| - Rank (r): 16 | |
| - Alpha: 16 | |
| - Target Modules: | |
| - q_proj (Query projection) | |
| - k_proj (Key projection) | |
| - v_proj (Value projection) | |
| - o_proj (Output projection) | |
| - up_proj (Upsampling projection) | |
| - down_proj (Downsampling projection) | |
| - gate_proj (Gate projection) | |
| - The order of loading adapters may affect the final result | |
| - Equal weights were chosen to maintain balanced influence from each adapter | |
| - The merged adapter maintains the same architecture and rank as the original adapters | |
| - While this adapter merges multiple fine-tunes, each component was developed as part of independent research efforts to explore and language model capabilities as part of R&D process. | |