Training Arguments

Overview

Qwen fine-tuning uses Hugging Face Transformers’ TrainingArguments with additional custom arguments for model loading, data processing, and LoRA training.

Argument Classes

Four argument classes configure different aspects of training:

from transformers import HfArgumentParser
from dataclasses import dataclass, field

parser = HfArgumentParser((
    ModelArguments,
    DataArguments,
    TrainingArguments,
    LoraArguments
))

model_args, data_args, training_args, lora_args = parser.parse_args_into_dataclasses()

ModelArguments

Specify which model to fine-tune.

model_name_or_path

str

default:"Qwen/Qwen-7B"

Hugging Face model ID or local path to model checkpoint:

--model_name_or_path Qwen/Qwen-7B-Chat

DataArguments

Configure training and evaluation data.

data_path

str

required

Path to training data JSON file:

--data_path ./data/train.json

eval_data_path

str

default:"None"

Path to evaluation data JSON file (optional):

--eval_data_path ./data/eval.json

lazy_preprocess

bool

default:"False"

Use lazy data loading to reduce memory usage:

--lazy_preprocess

Enable for very large datasets that don’t fit in memory.

TrainingArguments

Extends standard Hugging Face TrainingArguments with Qwen-specific options.

Core Training Parameters

output_dir

str

required

Directory for saving model checkpoints and outputs:

--output_dir ./output/qwen-finetuned

num_train_epochs

int

default:"3"

Number of training epochs:

--num_train_epochs 5

per_device_train_batch_size

int

default:"8"

Batch size per GPU during training:

--per_device_train_batch_size 4

per_device_eval_batch_size

int

default:"8"

Batch size per GPU during evaluation:

--per_device_eval_batch_size 8

gradient_accumulation_steps

int

default:"1"

Number of steps to accumulate gradients before updating:

--gradient_accumulation_steps 4

Effective batch size = batch_size × gradient_accumulation_steps × num_gpus

Learning Rate

learning_rate

float

default:"5e-5"

Initial learning rate:

--learning_rate 1e-4

lr_scheduler_type

str

default:"linear"

Learning rate schedule:

linear: Linear decay
cosine: Cosine annealing
constant: No decay

--lr_scheduler_type cosine

warmup_steps

int

default:"0"

Number of warmup steps:

--warmup_steps 100

warmup_ratio

float

default:"0.0"

Warmup ratio (alternative to warmup_steps):

--warmup_ratio 0.1

Optimization

optim

str

default:"adamw_torch"

Optimizer to use:

adamw_torch: PyTorch AdamW
adamw_hf: Hugging Face AdamW
adafactor: Adafactor (memory efficient)

--optim adamw_torch

weight_decay

float

default:"0.0"

Weight decay coefficient:

--weight_decay 0.01

adam_beta1

float

default:"0.9"

Adam beta1 parameter

adam_beta2

float

default:"0.999"

Adam beta2 parameter

max_grad_norm

float

default:"1.0"

Maximum gradient norm for clipping:

--max_grad_norm 1.0

Model Configuration

model_max_length

int

default:"8192"

Maximum sequence length (input + output):

--model_max_length 4096

use_lora

bool

default:"False"

Enable LoRA fine-tuning:

--use_lora

cache_dir

str

default:"None"

Directory for caching downloaded models:

--cache_dir ./cache

Checkpointing

save_strategy

str

default:"steps"

When to save checkpoints:

steps: Every save_steps
epoch: Every epoch
no: No saving

--save_strategy steps

save_steps

int

default:"500"

Save checkpoint every N steps:

--save_steps 200

save_total_limit

int

default:"None"

Maximum number of checkpoints to keep:

--save_total_limit 3

Evaluation

evaluation_strategy

str

default:"no"

When to run evaluation:

steps: Every eval_steps
epoch: Every epoch
no: No evaluation

--evaluation_strategy steps

eval_steps

int

default:"None"

Evaluate every N steps:

--eval_steps 100

Logging

logging_steps

int

default:"500"

Log metrics every N steps:

--logging_steps 10

logging_dir

str

default:"None"

TensorBoard log directory:

--logging_dir ./logs

report_to

str | list

default:"all"

Reporting integrations:

tensorboard
wandb
none

--report_to tensorboard

Performance

fp16

bool

default:"False"

Use FP16 mixed precision:

--fp16

bf16

bool

default:"False"

Use BF16 mixed precision (recommended for modern GPUs):

--bf16

gradient_checkpointing

bool

default:"False"

Enable gradient checkpointing to reduce memory:

--gradient_checkpointing

deepspeed

str

default:"None"

Path to DeepSpeed config file:

--deepspeed ./ds_config.json

Distributed Training

local_rank

int

default:"-1"

Local rank for distributed training (set automatically)

ddp_find_unused_parameters

bool

default:"False"

Find unused parameters in DDP:

--ddp_find_unused_parameters

Complete Example

python finetune.py \
  --model_name_or_path Qwen/Qwen-7B \
  --data_path ./data/train.json \
  --eval_data_path ./data/eval.json \
  --output_dir ./output/qwen-finetuned \
  --num_train_epochs 3 \
  --per_device_train_batch_size 2 \
  --per_device_eval_batch_size 2 \
  --gradient_accumulation_steps 8 \
  --evaluation_strategy steps \
  --eval_steps 100 \
  --save_strategy steps \
  --save_steps 200 \
  --save_total_limit 3 \
  --learning_rate 1e-4 \
  --weight_decay 0.01 \
  --warmup_ratio 0.03 \
  --lr_scheduler_type cosine \
  --logging_steps 10 \
  --model_max_length 2048 \
  --gradient_checkpointing \
  --bf16 \
  --deepspeed ds_config.json

Model API

OpenAI Compatible API

Training API

Training Arguments

Overview

Argument Classes

ModelArguments

DataArguments

TrainingArguments

Core Training Parameters

Learning Rate

Optimization

Model Configuration

Checkpointing

Evaluation

Logging

Performance

Distributed Training

Complete Example

Build docs developers (and LLMs) love

Model API

OpenAI Compatible API

Training API

​Overview

​Argument Classes

​ModelArguments

​DataArguments

​TrainingArguments

​Core Training Parameters

​Learning Rate

​Optimization

​Model Configuration

​Checkpointing

​Evaluation

​Logging

​Performance

​Distributed Training

​Complete Example

Build docs developers (and LLMs) love

Overview

Argument Classes

ModelArguments

DataArguments

TrainingArguments

Core Training Parameters

Learning Rate

Optimization

Model Configuration

Checkpointing

Evaluation

Logging

Performance

Distributed Training

Complete Example