I am currently following the fine-tuning methods for the Hugging Face model Zephyr 7B. They have implemented two fine-tuning methods, namely SFT and DPO, on a public dataset. Currently, I am fine-tuning a 7B model using SFT, which is progressing well. However, I have a question regarding whether it is acceptable to fine-tune the model on a DPO dataset generated synthetically by GPT-3.5.

From my understanding, DPO should be trained on the answers produced by the same model. I want to confirm this and inquire if anyone has attempted such fine-tuning before.

1 Answer 1


Using GPT-3.5-generated data for Domain-Adaptive Pretraining (DAPT) on a 7B model is an interesting idea, but it's typically best to use data generated by the same model you're fine-tuning. This ensures consistency and relevance. While there's no rule against using synthetic data from GPT-3.5, the success of such an approach would be experimental. If you decide to try it, make sure to thoroughly evaluate the quality of the synthetic data and compare the fine-tuning results with a baseline. It's also worth checking with the community to see if anyone else has shared experiences with a similar setup.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.