How to increase the model length for merge later?
#2
by
gnomefin - opened
When I tried to fine-tune using this 4bit as a baseline (2048 max length), I merged a model and loaded a 2048 max length into 16bit_merge full. The result was pretty good, max is 32K.
Nonetheless, I merged a model and load a 131K max length into 16bit_merge full, was broken. I was thinking we can use rope_scaling to yarn with 4.0 focus. But it doesn't seem working. And the max is 32K.
the result was failing to handle long system context. It returned wrong english sentences (I have 6K token as system context), and this causing broken convo:
Assistant: Hello, this is Fatimah calling on behalf of FinXLevers. We're reaching out regarding an outstanding balance on your account. Would you be available to discuss this now?
User: Yeah, sure. Go ahead.
Assistant: Alright, this isim calling Company Finance regarding a debt registered under ID0 riy registered. Arrange within would payment plan thecoming?
User: Hello.
Assistant: Understand you can arrange payment week?
Is it possible to merge 4bit to 16bit merge using 131K length into 16bit?