Update README.md

README.md

@@ -4,29 +4,79 @@ tags:
 - generated_from_trainer
 base_model: mistralai/Mixtral-8x7B-v0.1
 model-index:
-- name: workspace/volume/mixtral-limarp-qlora-out
+- name: limarp-zloss-mixtral-8x7b-qlora
   results: []
+datasets:
+- lemonilia/LimaRP
+language:
+- en
+license: apache-2.0
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
 [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
-# workspace/volume/mixtral-limarp-qlora-out
+# limarp-zloss-mixtral-8x7b-qlora
 
-This model was trained from scratch on the None dataset.
+Experimental limarp qlora trained at 10k ctx length (greater than size of the longest limarp sample when tokenized via mistral's tokenizer) on [mistralai/Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) using [Charles Goddard](https://huggingface.co/chargoddard)'s ZLoss and Megablocks-based fork of transformers.
 
 ## Model description
 
-More information needed
+The intended prompt format is the Alpaca instruction format of LimaRP v3:
+```
+### Instruction:
+Character's Persona: {bot character description}
+
+User's Persona: {user character description}
+
+Scenario: {what happens in the story}
+
+Play the role of Character. Taking the above information into consideration, you must engage in a roleplaying chat with User below this line. Do not write dialogues and narration for User.
+
+### Input:
+User: {utterance}
+
+### Response:
+Character: {utterance}
+
+### Input:
+User: {utterance}
+
+### Response:
+Character: {utterance}
+
+(etc.)
+```
+Inspired by the previously named "Roleplay" preset in SillyTavern, with this version of LimaRP it is possible to append a length modifier to the response instruction sequence, like this:
+
+```
+### Input
+User: {utterance}
+
+### Response: (length = medium)
+Character: {utterance}
+```
+
+This has an immediately noticeable effect on bot responses. The lengths using during training are:
+`micro`, `tiny`, `short`, `medium`, `long`, `massive`, `huge`, `enormous`, `humongous`, `unlimited`.
+**The recommended starting length is medium**. Keep in mind that the AI can ramble or impersonate
+the user with very long messages.
+
+The length control effect is reproducible, but the messages will not necessarily follow
+lengths very precisely, rather follow certain ranges on average, as seen in this table
+with data from tests made with one reply at the beginning of the conversation:
+
+![lengths](https://i.imgur.com/2WXGgaV.png)
+
+Response length control appears to work well also deep into the conversation. **By omitting
+the modifier, the model will choose the most appropriate response length** (although it might
+not necessarily be what the user desires).
 
 ## Intended uses & limitations
 
-More information needed
+The model will show biases similar to those observed in niche roleplaying forums on the Internet, besides those exhibited by the base model.
 
 ## Training and evaluation data
 
-More information needed
+For more details about LimaRP, see the dataset page.
 
 ## Training procedure
 
@@ -44,10 +94,6 @@ The following hyperparameters were used during training:
 - lr_scheduler_warmup_steps: 10
 - num_epochs: 3
 
-### Training results
-
-
-
 ### Framework versions
 
 - Transformers 4.37.0.dev0
@@ -72,4 +118,4 @@ The following `bitsandbytes` quantization config was used during training:
 ### Framework versions
 
 
-- PEFT 0.6.0
+- PEFT 0.6.0