{"id":111701,"date":"2026-10-01T16:25:05","date_gmt":"2026-10-01T08:25:05","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/111701.html"},"modified":"2026-10-01T16:25:05","modified_gmt":"2026-10-01T08:25:05","slug":"llava1-5-7b%e5%a4%8d%e7%8e%b0","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/111701.html","title":{"rendered":"LLaVA1.5-7B\u590d\u73b0"},"content":{"rendered":"<\/p>\n<h4>\u6587\u7ae0\u76ee\u5f55<\/h4>\n<ul>\n<li>\u524d\u8a00<\/li>\n<li>\u4e00\u3001\u539f\u7406&#xff1a;\u4e09\u5757\u79ef\u6728\u62fc\u51fa\u6765\u7684\u591a\u6a21\u6001<\/li>\n<li>\n<ul>\n<li>1. \u7ed3\u6784&#xff1a;LLM &#043; Align Layer &#043; ViT<\/li>\n<li>2. \u635f\u5931\u8ba1\u7b97<\/li>\n<\/ul>\n<\/li>\n<li>\u4e8c\u3001\u8bba\u6587\u91cc\u7684\u5b9e\u9a8c<\/li>\n<li>\u4e09\u3001\u590d\u73b0&#xff1a;8\u00d7L20 \u4e0a\u7684\u53cc\u9636\u6bb5\u8bad\u7ec3<\/li>\n<li>\n<ul>\n<li>1. stage1&#xff1a;Align \u5c42\u5bf9\u9f50\u8bad\u7ec3<\/li>\n<li>2. stage2&#xff1a;Visual \u6307\u4ee4\u5fae\u8c03<\/li>\n<li>3. \u5b9e\u9a8c\u7ed3\u679c&#xff1a;POPE \u548c MME<\/li>\n<\/ul>\n<\/li>\n<li>\u603b\u7ed3<\/li>\n<\/ul>\n<hr \/>\n<h2>\u524d\u8a00<\/h2>\n<p>\u2003\u2003LLaVA \u662f\u5f00\u6e90\u591a\u6a21\u6001\u5927\u6a21\u578b\u91cc\u6700\u597d\u4e0a\u624b\u7684\u9879\u76ee\u4e4b\u4e00\u3002<br \/>\n\u2003\u2003\u5b83\u628a\u89c6\u89c9\u7f16\u7801\u5668\u548c\u8bed\u8a00\u6a21\u578b\u62fc\u5728\u4e00\u8d77&#xff0c;\u5206\u4e24\u9636\u6bb5\u8bad\u7ec3\u3002<br \/>\n\u2003\u2003\u8bba\u6587\u548c\u4ee3\u7801\u90fd\u516c\u5f00&#xff0c;\u53ef\u771f\u5728 8 \u5361\u673a\u5668\u4e0a\u8dd1\u4e00\u904d&#xff0c;\u5751\u4e00\u70b9\u4e0d\u5c11\u3002<br \/>\n\u2003\u2003\u672c\u6587\u5148\u8bb2\u7ed3\u6784&#xff0c;\u518d\u770b\u8bba\u6587\u5b9e\u9a8c&#xff0c;\u6700\u540e\u7ed9\u51fa 8\u00d7L20 \u4e0a\u7684\u590d\u73b0\u811a\u672c\u548c\u8e29\u5751\u8bb0\u5f55\u3002<br \/>\n\u2003\u2003\u9002\u5408\u60f3\u81ea\u5df1\u8dd1\u4e00\u904d LLaVA \u7684\u540c\u5b66\u3002<\/p>\n<p>\u8bba\u6587: https:\/\/arxiv.org\/abs\/2310.03744<br \/>\n\u4ee3\u7801: https:\/\/github.com\/haotian-liu\/LLaVA\/tree\/main<\/p>\n<hr \/>\n<h2>\u4e00\u3001\u539f\u7406&#xff1a;\u4e09\u5757\u79ef\u6728\u62fc\u51fa\u6765\u7684\u591a\u6a21\u6001<\/h2>\n<h3>1. \u7ed3\u6784&#xff1a;LLM &#043; Align Layer &#043; ViT<\/h3>\n<p>\u2003\u2003LLaVA \u7684\u7ed3\u6784\u53ea\u6709\u4e09\u5757&#xff1a;LLM\u3001Align Layer\u3001ViT\u3002<br \/>\n\u2003\u2003ViT \u8d1f\u8d23\u770b\u56fe\u3002\u5b83\u628a\u56fe\u7247\u5207\u6210 patch&#xff0c;\u7f16\u7801\u6210\u4e00\u4e32\u89c6\u89c9\u7279\u5f81\u3002<br \/>\n\u2003\u2003Align Layer \u662f\u4e24\u5c42 MLP&#xff0c;\u628a\u89c6\u89c9\u7279\u5f81\u6295\u5f71\u5230\u8bed\u8a00\u6a21\u578b\u7684\u8bcd\u5411\u91cf\u7a7a\u95f4\u3002<br \/>\n\u2003\u2003LLM \u8d1f\u8d23\u7406\u89e3\u3002\u5b83\u62ff\u5230\u89c6\u89c9 token&#xff0c;\u548c\u6587\u5b57 token \u4e00\u8d77\u505a\u81ea\u56de\u5f52\u751f\u6210\u3002<br \/>\n\u2003\u2003\u771f\u6b63\u8981\u5b66\u7684\u53ea\u6709\u4e2d\u95f4\u90a3\u5c42 MLP\u3002stage-1 \u53ea\u8bad\u5b83&#xff0c;\u6536\u655b\u5f88\u5feb\u3002<br \/>\n\u2003\u2003\u7ed3\u6784\u56fe\u5982\u4e0b\u3002<br \/>\n\u2003\u2003<img decoding=\"async\" src=\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/10\/20261001082503-6abe18dfd14fc.png\" alt=\"\u5728\u8fd9\u91cc\u63d2\u5165\u56fe\u7247\u63cf\u8ff0\" \/><br \/>\n\u3002<\/p>\n<h3>2. \u635f\u5931\u8ba1\u7b97<\/h3>\n<p>\u2003\u2003\u8bad\u7ec3\u65f6\u4e0d\u662f\u6240\u6709 token \u90fd\u7b97\u635f\u5931\u3002<br \/>\n\u2003\u2003\u8f93\u5165\u91cc\u6709\u56fe\u7247 token\u3001\u7528\u6237\u63d0\u95ee\u3001\u6a21\u578b\u56de\u7b54\u3002<br \/>\n\u2003\u2003\u4ea4\u53c9\u71b5\u635f\u5931\u53ea\u4f5c\u7528\u5728\u7eff\u8272\u7684\u56de\u7b54\u90e8\u5206&#xff0c;\u56fe\u7247 token \u548c\u63d0\u95ee\u90fd\u88ab mask \u6389\u3002<br \/>\n\u2003\u2003\u6211\u4eec\u53ea\u60f3\u8ba9\u6a21\u578b\u5b66\u4f1a\u600e\u4e48\u7b54&#xff0c;\u4e0d\u60f3\u8ba9\u5b83\u9884\u6d4b\u7528\u6237\u4f1a\u95ee\u4ec0\u4e48\u3002<br \/>\n\u2003\u2003\u4e0b\u56fe\u7eff\u8272\u5b57\u4f53\u624d\u7b97\u635f\u5931\u3002<br \/>\n\u2003\u2003<img decoding=\"async\" src=\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/10\/20261001082504-6abe18e01aac7.png\" alt=\"\u5728\u8fd9\u91cc\u63d2\u5165\u56fe\u7247\u63cf\u8ff0\" \/><br \/>\n\u3002<\/p>\n<hr \/>\n<h2>\u4e8c\u3001\u8bba\u6587\u91cc\u7684\u5b9e\u9a8c<\/h2>\n<p>\u2003\u2003\u8bba\u6587\u5728 11 \u4e2a\u57fa\u51c6\u4e0a\u505a\u4e86\u8bc4\u6d4b&#xff0c;\u8986\u76d6\u95ee\u7b54\u3001OCR\u3001\u63a8\u7406\u7b49\u65b9\u5411\u3002<br \/>\n\u2003\u2003\u6570\u636e\u53ea\u7528 1.2M&#xff0c;\u6bd4\u540c\u671f\u65b9\u6848\u5c0f\u4e00\u4e2a\u91cf\u7ea7&#xff0c;\u6548\u679c\u66f4\u597d\u3002<br \/>\n\u2003\u2003\u5176\u4e2d POPE \u4e13\u95e8\u6d4b\u5e7b\u89c9&#xff0c;MME \u6d4b\u611f\u77e5\u548c\u8ba4\u77e5&#xff0c;\u662f\u590d\u73b0\u65f6\u6700\u5e38\u770b\u7684\u4e24\u9879\u3002<br \/>\n\u2003\u2003\u7ed3\u8bba\u5f88\u6e05\u695a&#xff1a;\u7ed3\u6784\u7b80\u5355\u52a0\u6570\u636e\u5e72\u51c0&#xff0c;\u6bd4\u5806\u6a21\u5757\u7ba1\u7528\u3002<br \/>\n\u2003\u2003\u5bf9\u6bd4\u7ed3\u679c\u5982\u4e0b\u3002<br \/>\n\u2003\u2003<img decoding=\"async\" src=\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/10\/20261001082504-6abe18e04a9af.png\" alt=\"\u5728\u8fd9\u91cc\u63d2\u5165\u56fe\u7247\u63cf\u8ff0\" \/><\/p>\n<h2>\u4e09\u3001\u590d\u73b0&#xff1a;8\u00d7L20 \u4e0a\u7684\u53cc\u9636\u6bb5\u8bad\u7ec3<\/h2>\n<p>\u2003\u2003\u6211\u7528\u7684\u673a\u5668\u662f 8 \u5f20 L20&#xff0c;\u5355\u5361 46GB \u663e\u5b58\u3002<br \/>\n\u2003\u2003\u6574\u4e2a\u6d41\u7a0b\u5206\u4e24\u6b65&#xff1a;stage-1 \u5bf9\u9f50&#xff0c;stage-2 \u6307\u4ee4\u5fae\u8c03\u3002<\/p>\n<h3>1. stage1&#xff1a;Align \u5c42\u5bf9\u9f50\u8bad\u7ec3<\/h3>\n<p>\u2003\u2003\u8fd9\u4e00\u6b65\u53ea\u8bad Align Layer&#xff0c;ViT \u548c LLM \u5168\u90e8\u51bb\u7ed3\u3002<br \/>\n\u2003\u2003\u6570\u636e\u7528 558K \u56fe\u6587\u5bf9&#xff0c;\u8ba9\u89c6\u89c9\u7279\u5f81\u5b66\u4f1a\u8bf4\u4eba\u8bdd\u3002<br \/>\n\u2003\u2003\u6709\u51e0\u4e2a\u53c2\u6570\u548c\u5b98\u65b9 A100 \u811a\u672c\u4e0d\u540c&#xff0c;\u90fd\u662f\u663e\u5b58\u903c\u51fa\u6765\u7684&#xff1a;<\/p>\n<ul>\n<li>\u6a21\u578b\u6362\u6210 7B\u3002ZeRO-2 \u4e0d\u5207\u5206\u53c2\u6570&#xff0c;\u6bcf\u5361\u4e00\u4efd\u5b8c\u6574\u526f\u672c\u300213B \u7684 bf16 \u8981 26GB&#xff0c;46GB \u4f59\u91cf\u592a\u5c11\u3002<\/li>\n<li>\u5168\u5c40 batch \u5bf9\u9f50\u8bba\u6587\u3002\u5355\u5361 16&#xff0c;\u7d2f\u79ef 2 \u6b65&#xff0c;8 \u5361\u5408\u8ba1 256\u3002<\/li>\n<li>\u6bcf 1000 \u6b65\u5b58\u4e00\u6b21\u3002\u4e00\u4e2a epoch \u6709 2180 \u6b65&#xff0c;\u5d29\u4e86\u4e0d\u81f3\u4e8e\u5168\u767d\u8dd1\u3002<\/li>\n<li>\u5173\u6389 wandb \u4e0a\u62a5\u3002\u5b83\u88c5\u4e86\u4f46\u6ca1\u767b\u5f55&#xff0c;\u521d\u59cb\u5316\u4f1a\u5361\u5728\u8981 key \u7684\u63d0\u793a\u4e0a\u3002<\/li>\n<\/ul>\n<p>\u2003\u2003\u811a\u672c\u5f00\u5934\u52a0\u4e86\u9884\u68c0&#xff0c;\u786e\u8ba4 558K \u56fe\u7247\u89e3\u538b\u5b8c\u6574\u3002\u7f3a\u5931\u65f6 dataloader \u4f1a\u5728\u4e2d\u9014\u624d\u62a5\u9519\u3002<\/p>\n<p><span class=\"token shebang important\">#!\/bin\/bash<\/span><br \/>\n<span class=\"token comment\"># Stage 1 (feature alignment \/ pretrain) for LLaVA-v1.5 on 8x NVIDIA L20 (46GB).<\/span><br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token comment\"># Differences from scripts\/v1_5\/pretrain.sh (the 8xA100 reference):<\/span><br \/>\n<span class=\"token comment\">#   &#8211; 7B Vicuna instead of 13B: ZeRO-2 does not shard parameters, so every GPU<\/span><br \/>\n<span class=\"token comment\">#     holds a full replica. 13B bf16 &#061; 26GB\/card leaves too little headroom on 46GB.<\/span><br \/>\n<span class=\"token comment\">#   &#8211; batch 16 x accum 2 x 8 GPUs &#061; 256 global, matching the paper&#039;s global batch.<\/span><br \/>\n<span class=\"token comment\">#   &#8211; checkpointing every 1000 steps so a crash doesn&#039;t cost the whole epoch<\/span><br \/>\n<span class=\"token comment\">#     (2180 steps\/epoch at global batch 256).<\/span><br \/>\n<span class=\"token comment\">#   &#8211; &#8211;report_to none: wandb is installed but unauthenticated, and &#096;wandb.init()&#096;<\/span><br \/>\n<span class=\"token comment\">#     blocks on an API-key prompt. Set to &#034;wandb&#034; once you have run &#096;wandb login&#096;.<\/span><br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token comment\"># Run:  conda activate org_llava &amp;&amp; bash scripts\/v1_5\/pretrain_8xL20.sh<\/span><br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token builtin class-name\">set<\/span> <span class=\"token parameter variable\">-euo<\/span> pipefail<br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token builtin class-name\">export<\/span> <span class=\"token assign-left variable\">CUDA_VISIBLE_DEVICES<\/span><span class=\"token operator\">&#061;<\/span><span class=\"token number\">0,1<\/span>,2,3,4,5,6,7<br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token assign-left variable\">DATA_JSON<\/span><span class=\"token operator\">&#061;<\/span>.\/playground\/data\/LLaVA-Pretrain\/blip_laion_cc_sbu_558k.json<br \/>\n<span class=\"token assign-left variable\">IMAGE_DIR<\/span><span class=\"token operator\">&#061;<\/span>.\/playground\/data\/LLaVA-Pretrain\/images<br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token comment\"># Local copies of both hub checkpoints, so nothing is fetched mid-run and a container<\/span><br \/>\n<span class=\"token comment\"># restart can&#039;t pull the weights out from under a training job.<\/span><br \/>\n<span class=\"token assign-left variable\">MODEL_PATH<\/span><span class=\"token operator\">&#061;<\/span>.\/pretrain_weight\/vicuna-7b-v1.5<br \/>\n<span class=\"token assign-left variable\">VISION_TOWER<\/span><span class=\"token operator\">&#061;<\/span>.\/checkpoints\/hf_models\/clip-vit-large-patch14-336<br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token comment\"># &#8212; preflight: the 558k images must be fully extracted before training starts &#8212;<\/span><br \/>\n<span class=\"token keyword\">if<\/span> <span class=\"token punctuation\">[<\/span> <span class=\"token operator\">!<\/span> <span class=\"token parameter variable\">-f<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$DATA_JSON<\/span>&#034;<\/span> <span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">;<\/span> <span class=\"token keyword\">then<\/span><br \/>\n    <span class=\"token builtin class-name\">echo<\/span> <span class=\"token string\">&#034;ERROR: missing <span class=\"token variable\">$DATA_JSON<\/span>&#034;<\/span> <span class=\"token operator\">&gt;<\/span><span class=\"token file-descriptor important\">&amp;2<\/span><br \/>\n    <span class=\"token builtin class-name\">exit<\/span> <span class=\"token number\">1<\/span><br \/>\n<span class=\"token keyword\">fi<\/span><br \/>\n<span class=\"token keyword\">for<\/span> <span class=\"token for-or-select variable\">p<\/span> <span class=\"token keyword\">in<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$MODEL_PATH<\/span>&#034;<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$VISION_TOWER<\/span>&#034;<\/span><span class=\"token punctuation\">;<\/span> <span class=\"token keyword\">do<\/span><br \/>\n    <span class=\"token keyword\">if<\/span> <span class=\"token punctuation\">[<\/span> <span class=\"token operator\">!<\/span> <span class=\"token parameter variable\">-d<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$p<\/span>&#034;<\/span> <span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">;<\/span> <span class=\"token keyword\">then<\/span><br \/>\n        <span class=\"token builtin class-name\">echo<\/span> <span class=\"token string\">&#034;ERROR: missing <span class=\"token variable\">$p<\/span> (run from the repo root)&#034;<\/span> <span class=\"token operator\">&gt;<\/span><span class=\"token file-descriptor important\">&amp;2<\/span><br \/>\n        <span class=\"token builtin class-name\">exit<\/span> <span class=\"token number\">1<\/span><br \/>\n    <span class=\"token keyword\">fi<\/span><br \/>\n<span class=\"token keyword\">done<\/span><br \/>\n<span class=\"token assign-left variable\">N_IMAGES<\/span><span class=\"token operator\">&#061;<\/span><span class=\"token variable\"><span class=\"token variable\">$(<\/span><span class=\"token function\">find<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$IMAGE_DIR<\/span>&#034;<\/span> <span class=\"token parameter variable\">-name<\/span> <span class=\"token string\">&#039;*.jpg&#039;<\/span> <span class=\"token operator\"><span class=\"token file-descriptor important\">2<\/span>&gt;<\/span>\/dev\/null <span class=\"token operator\">|<\/span> <span class=\"token function\">wc<\/span> <span class=\"token parameter variable\">-l<\/span><span class=\"token variable\">)<\/span><\/span><br \/>\n<span class=\"token builtin class-name\">echo<\/span> <span class=\"token string\">&#034;images found: <span class=\"token variable\">$N_IMAGES<\/span> \/ 558000&#034;<\/span><br \/>\n<span class=\"token keyword\">if<\/span> <span class=\"token punctuation\">[<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$N_IMAGES<\/span>&#034;<\/span> <span class=\"token parameter variable\">-lt<\/span> <span class=\"token number\">550000<\/span> <span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">;<\/span> <span class=\"token keyword\">then<\/span><br \/>\n    <span class=\"token builtin class-name\">echo<\/span> <span class=\"token string\">&#034;WARNING: extraction looks incomplete; the dataloader will crash on missing images.&#034;<\/span> <span class=\"token operator\">&gt;<\/span><span class=\"token file-descriptor important\">&amp;2<\/span><br \/>\n<span class=\"token keyword\">fi<\/span><br \/>\n<span class=\"token comment\">#<\/span><br \/>\ndeepspeed llava\/train\/train_mem.py <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;deepspeed<\/span> .\/scripts\/zero2.json <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;model_name_or_path<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$MODEL_PATH<\/span>&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;version<\/span> plain <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;data_path<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$DATA_JSON<\/span>&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;image_folder<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$IMAGE_DIR<\/span>&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;vision_tower<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$VISION_TOWER<\/span>&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;mm_projector_type<\/span> mlp2x_gelu <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;tune_mm_mlp_adapter<\/span> True <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;mm_vision_select_layer<\/span> <span class=\"token parameter variable\">-2<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;mm_use_im_start_end<\/span> False <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;mm_use_im_patch_token<\/span> False <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;bf16<\/span> True <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;output_dir<\/span> .\/checkpoints\/llava-v1.5-7b-pretrain <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;num_train_epochs<\/span> <span class=\"token number\">1<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;per_device_train_batch_size<\/span> <span class=\"token number\">16<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;per_device_eval_batch_size<\/span> <span class=\"token number\">4<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;gradient_accumulation_steps<\/span> <span class=\"token number\">2<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;evaluation_strategy<\/span> <span class=\"token string\">&#034;no&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;save_strategy<\/span> <span class=\"token string\">&#034;steps&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;save_steps<\/span> <span class=\"token number\">1000<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;save_total_limit<\/span> <span class=\"token number\">2<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;learning_rate<\/span> 1e-3 <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;weight_decay<\/span> <span class=\"token number\">0<\/span>. <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;warmup_ratio<\/span> <span class=\"token number\">0.03<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;lr_scheduler_type<\/span> <span class=\"token string\">&#034;cosine&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;logging_steps<\/span> <span class=\"token number\">1<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;tf32<\/span> True <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;model_max_length<\/span> <span class=\"token number\">2048<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;gradient_checkpointing<\/span> True <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;dataloader_num_workers<\/span> <span class=\"token number\">4<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;lazy_preprocess<\/span> True <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;report_to<\/span> none<\/p>\n<h3>2. stage2&#xff1a;Visual \u6307\u4ee4\u5fae\u8c03<\/h3>\n<p>\u2003\u2003\u8fd9\u4e00\u6b65\u8981\u66f4\u65b0\u5168\u90e8 7B \u53c2\u6570&#xff0c;\u8bad\u7ec3\u7b56\u7565\u5fc5\u987b\u6362\u3002<br \/>\n\u2003\u2003\u6700\u8981\u7d27\u7684\u662f\u6362 DeepSpeed \u914d\u7f6e&#xff0c;stage-1 \u7528 zero2.json&#xff0c;\u8fd9\u91cc\u6362\u6210 zero3.json\u3002<br \/>\n\u2003\u2003ZeRO-2 \u4e0d\u5207\u5206\u53c2\u6570&#xff0c;\u6bcf\u5361\u653e\u4e0d\u4e0b 7B&#xff0c;\u53ea\u80fd\u8ba9 ZeRO-3 \u628a\u53c2\u6570\u4e5f\u5207\u4e86\u3002<br \/>\n\u2003\u2003\u5176\u4f59\u5173\u952e\u5dee\u5f02\u6709\u8fd9\u51e0\u5904&#xff1a;<\/p>\n<ul>\n<li>\u7248\u672c\u6539\u6210 v1&#xff0c;\u542f\u7528\u5bf9\u8bdd\u6a21\u677f\u3002stage-1 \u662f plain\u3002<\/li>\n<li>\u6570\u636e\u6362\u6210 665K \u6df7\u5408\u6307\u4ee4\u96c6\u3002<\/li>\n<li>\u56fe\u7247\u6839\u76ee\u5f55\u6307\u5230 data \u8fd9\u4e00\u5c42&#xff0c;json \u91cc\u662f\u76f8\u5bf9\u8def\u5f84\u3002<\/li>\n<li>\u6ce8\u5165 stage-1 \u8bad\u597d\u7684\u6295\u5f71\u5c42\u3002<\/li>\n<li>\u53bb\u6389\u53ea\u8bad adapter \u7684\u5f00\u5173&#xff0c;\u53d8\u6210\u5168\u91cf\u5fae\u8c03\u3002<\/li>\n<li>\u5b66\u4e60\u7387 2e-5&#xff0c;\u6bd4 stage-1 \u4f4e 50 \u500d\u3002<\/li>\n<li>\u52a0\u4e0a\u6309\u6bd4\u4f8b\u586b\u5145\u3001\u6309\u957f\u5ea6\u5206\u7ec4&#xff0c;1.5 \u7248\u6807\u914d\u3002<\/li>\n<\/ul>\n<p>\u2003\u2003\u5168\u91cf\u5fae\u8c03\u6709\u4e2a\u526f\u4f5c\u7528&#xff1a;checkpoint \u4f1a\u5f88\u5927\u3002<br \/>\n\u2003\u2003\u5355\u4e2a checkpoint \u7ea6 88GB\u3002bf16 \u6743\u91cd 12.6GB&#xff0c;fp32 master 25.1GB&#xff0c;Adam \u52a8\u91cf 50.2GB\u3002<br \/>\n\u2003\u2003\u4f18\u5316\u5668\u72b6\u6001\u5360\u4e86\u516d\u5206\u4e4b\u4e03\u3002<br \/>\n\u2003\u2003\u53ea\u6309 7B \u7684 14GB \u53bb\u4f30&#xff0c;\u4f1a\u5c11\u7b97 6 \u500d&#xff0c;2026 \u5e74 9 \u6708 22 \u53f7\u6211\u8e29\u8fc7\u8fd9\u4e2a\u5751\u3002<br \/>\n\u2003\u2003\u4fdd\u5b58\u95f4\u9694\u4e5f\u662f\u8840\u6cea\u6559\u8bad\u3002\u4e0a\u6e38\u9ed8\u8ba4 1000 \u6b65\u5b58\u4e00\u6b21\u3002<br \/>\n\u2003\u2003\u7ed3\u679c\u7b2c\u4e00\u6b21\u4fdd\u5b58\u5c31\u5931\u8d25&#xff0c;\u767d\u8dd1 4 \u5c0f\u65f6 08 \u5206\u3002<br \/>\n\u2003\u2003\u6240\u4ee5\u6211\u628a\u95f4\u9694\u6539\u6210 500 \u6b65\u3002\u5168\u7a0b\u5b58 10 \u6b21&#xff0c;\u591a\u82b1\u7ea6 1 \u5c0f\u65f6\u5199\u76d8&#xff0c;\u6362\u6765\u5d29\u4e00\u6b21\u6700\u591a\u4e22 2 \u5c0f\u65f6\u3002<br \/>\n\u2003\u2003\u5199\u76d8\u6162\u662f\u786c\u4ef6\u51b3\u5b9a\u7684\u3002NFS \u5b9e\u6d4b 249MB\/s&#xff0c;\u4e00\u6b21\u4fdd\u5b58\u8981 6 \u5206\u949f\u3002<br \/>\n\u2003\u2003\u53ea\u8981\u4e2d\u95f4\u4ea7\u7269\u505a\u8bc4\u6d4b\u3001\u4e0d\u9700\u8981\u7eed\u8bad&#xff0c;\u5c31\u6253\u5f00\u53ea\u5b58\u6743\u91cd\u7684\u5f00\u5173\u3002checkpoint \u4f1a\u7626\u8eab&#xff1a;88GB \u964d\u5230 12.6GB\u3002<\/p>\n<p><span class=\"token shebang important\">#!\/bin\/bash<\/span><br \/>\n<span class=\"token comment\"># Stage 2 (visual instruction tuning) for LLaVA-v1.5-7B on 8x NVIDIA L20 (46GB).<\/span><br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token comment\"># \u63a5 stage-1 \u7684\u4ea7\u7269:&#8211;pretrain_mm_mlp_adapter \u6307\u5411 checkpoints\/llava-v1.5-7b-pretrain\/mm_projector.bin<\/span><br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token comment\"># \u548c pretrain_8xL20.sh \u7684 8 \u5904\u5173\u952e\u5dee\u5f02:<\/span><br \/>\n<span class=\"token comment\">#   &#8211; deepspeed \u6362\u6210 zero3.json: stage-2 \u8981\u66f4\u65b0\u5168\u90e8 7B \u53c2\u6570, ZeRO-2 \u4e0d\u5207\u5206\u53c2\u6570,<\/span><br \/>\n<span class=\"token comment\">#     \u6bcf\u5361\u4e00\u4efd 7B \u653e\u4e0d\u4e0b, \u5fc5\u987b\u8ba9 ZeRO-3 \u628a\u53c2\u6570\u4e5f\u5207\u4e86\u3002<\/span><br \/>\n<span class=\"token comment\">#   &#8211; &#8211;version v1 (stage-1 \u662f plain): \u542f\u7528 vicuna \u5bf9\u8bdd\u6a21\u677f\u3002<\/span><br \/>\n<span class=\"token comment\">#   &#8211; &#8211;data_path \u6362\u6210 llava_v1_5_mix665k.json\u3002<\/span><br \/>\n<span class=\"token comment\">#   &#8211; &#8211;image_folder \u6307\u5230 playground\/data \u8fd9\u4e00\u5c42: json \u91cc\u7684\u8def\u5f84\u662f<\/span><br \/>\n<span class=\"token comment\">#     &#034;coco\/train2017\/xxx.jpg&#034; \u8fd9\u79cd\u76f8\u5bf9\u8def\u5f84, \u4e0d\u662f\u76f8\u5bf9 coco\/\u3002<\/span><br \/>\n<span class=\"token comment\">#   &#8211; &#8211;pretrain_mm_mlp_adapter \u6ce8\u5165 stage-1 \u8bad\u597d\u7684 projector\u3002<\/span><br \/>\n<span class=\"token comment\">#   &#8211; \u53bb\u6389 &#8211;tune_mm_mlp_adapter: \u4e0d\u5199\u5373 False, \u53d8\u6210\u5168\u91cf\u5fae\u8c03\u3002<\/span><br \/>\n<span class=\"token comment\">#     \u526f\u4f5c\u7528: LLaVATrainer._save_checkpoint \u4e0d\u518d\u8d70&#034;\u53ea\u5b58 adapter&#034;\u7684\u5206\u652f, \u800c\u662f\u8d70 HF \u7684<\/span><br \/>\n<span class=\"token comment\">#     save_pretrained, \u4e8e\u662f DeepSpeed \u7684\u4f18\u5316\u5668\u72b6\u6001\u4e5f\u4e00\u8d77\u843d\u76d8\u3002\u5355\u4e2a checkpoint \u2248 88GB<\/span><br \/>\n<span class=\"token comment\">#     (6.738B \u53c2\u6570\u5b9e\u6d4b\u7b97\u5f97): bf16 \u6743\u91cd 12.6GB &#043; fp32 master 25.1GB &#043; Adam exp_avg\/sq 50.2GB\u3002<\/span><br \/>\n<span class=\"token comment\">#     \u4f18\u5316\u5668\u72b6\u6001\u5360\u4e86 6\/7 \u2014\u2014 \u53ea\u6309&#034;\u6574\u4e2a 7B ~14GB&#034;\u4f30\u4f1a\u5c11\u7b97 6 \u500d(2026-09-22 \u8e29\u8fc7)\u3002<\/span><br \/>\n<span class=\"token comment\">#     save_total_limit 2 \u5cf0\u503c ~176GB, \u76d8\u4e0a\u8fd8\u6709 5TB, \u4e0d\u662f\u74f6\u9888; \u74f6\u9888\u662f\u5199\u76d8\u65f6\u95f4 \u2014\u2014<\/span><br \/>\n<span class=\"token comment\">#     NFS \u5b9e\u6d4b 249MB\/s, \u4e00\u6b21\u4fdd\u5b58\u7ea6 6 \u5206\u949f, \u8bad\u5230\u4e00\u534a\u4f1a\u56e0\u4e3a\u4fdd\u5b58\u800c\u505c\u987f\u3002<\/span><br \/>\n<span class=\"token comment\">#   &#8211; &#8211;image_aspect_ratio pad \/ &#8211;group_by_modality_length True: LLaVA-1.5 \u7684\u89c4\u5b9a\u3002<\/span><br \/>\n<span class=\"token comment\">#   &#8211; &#8211;learning_rate 2e-5 (stage-1 \u662f 1e-3, \u5dee 50 \u500d)\u3002<\/span><br \/>\n<span class=\"token comment\">#   &#8211; &#8211;save_steps 500 (\u4e0a\u6e38\u9ed8\u8ba4 1000): 5198 \u6b65\u5168\u7a0b\u4fdd\u5b58 10 \u6b21, \u591a\u82b1\u7ea6 1 \u5c0f\u65f6\u5199\u76d8\u3002<\/span><br \/>\n<span class=\"token comment\">#     \u6362\u6765\u7684\u662f\u4e00\u6761\u786c\u5e95\u7ebf \u2014\u2014 \u5d29\u4e00\u6b21\u6700\u591a\u4e22 2 \u5c0f\u65f6\u30022026-09-22 \u56e0\u4e3a save_steps&#061;1000 \u4e14\u9996\u6b21<\/span><br \/>\n<span class=\"token comment\">#     \u4fdd\u5b58\u5728\u7b2c 1000 \u6b65\u4fdd\u5b58\u5931\u8d25, \u76f4\u63a5\u767d\u8dd1 4 \u5c0f\u65f6 08 \u5206(\u6743\u91cd\u96f6\u5b57\u8282\u843d\u76d8)\u3002<\/span><br \/>\n<span class=\"token comment\">#     \u6ce8\u610f: \u53ea\u8981 &#8211;save_only_model \u6ca1\u5f00, \u6bcf\u4e2a checkpoint \u5c31\u5fc5\u987b\u5e26\u4f18\u5316\u5668\u72b6\u6001, \u5426\u5219\u65e0\u6cd5 resume<\/span><br \/>\n<span class=\"token comment\">#     (train.py \u89c1\u5230 checkpoint-* \u4f1a\u8d70 resume_from_checkpoint&#061;True)\u3002\u82e5\u53ea\u8981\u4e2d\u95f4\u4ea7\u7269\u505a\u8bc4\u6d4b\u3001<\/span><br \/>\n<span class=\"token comment\">#     \u4e0d\u9700\u8981\u65ad\u70b9\u7eed\u8bad, \u53ef\u52a0 &#8211;save_only_model True: checkpoint \u4ece 88GB \u964d\u5230 12.6GB, \u4fdd\u5b58\u4e5f\u5feb\u5f97\u591a\u3002<\/span><br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token comment\"># Run:  conda activate org_llava &amp;&amp; bash scripts\/v1_5\/finetune_8xL20.sh<\/span><br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token builtin class-name\">set<\/span> <span class=\"token parameter variable\">-euo<\/span> pipefail<br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token builtin class-name\">export<\/span> <span class=\"token assign-left variable\">CUDA_VISIBLE_DEVICES<\/span><span class=\"token operator\">&#061;<\/span><span class=\"token number\">0,1<\/span>,2,3,4,5,6,7<br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token assign-left variable\">DATA_JSON<\/span><span class=\"token operator\">&#061;<\/span>.\/playground\/data\/llava_v1_5_mix665k.json<br \/>\n<span class=\"token assign-left variable\">IMAGE_DIR<\/span><span class=\"token operator\">&#061;<\/span>.\/playground\/data<br \/>\n<span class=\"token assign-left variable\">MODEL_PATH<\/span><span class=\"token operator\">&#061;<\/span>.\/pretrain_weight\/vicuna-7b-v1.5<br \/>\n<span class=\"token assign-left variable\">VISION_TOWER<\/span><span class=\"token operator\">&#061;<\/span>.\/checkpoints\/hf_models\/clip-vit-large-patch14-336<br \/>\n<span class=\"token assign-left variable\">MM_PROJECTOR<\/span><span class=\"token operator\">&#061;<\/span>.\/checkpoints\/llava-v1.5-7b-pretrain\/mm_projector.bin<br \/>\n<span class=\"token assign-left variable\">OUTPUT_DIR<\/span><span class=\"token operator\">&#061;<\/span>.\/checkpoints\/llava-v1.5-7b<br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token comment\"># &#8212; preflight &#8212;<\/span><br \/>\n<span class=\"token keyword\">for<\/span> <span class=\"token for-or-select variable\">f<\/span> <span class=\"token keyword\">in<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$DATA_JSON<\/span>&#034;<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$MM_PROJECTOR<\/span>&#034;<\/span><span class=\"token punctuation\">;<\/span> <span class=\"token keyword\">do<\/span><br \/>\n    <span class=\"token keyword\">if<\/span> <span class=\"token punctuation\">[<\/span> <span class=\"token operator\">!<\/span> <span class=\"token parameter variable\">-f<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$f<\/span>&#034;<\/span> <span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">;<\/span> <span class=\"token keyword\">then<\/span><br \/>\n        <span class=\"token builtin class-name\">echo<\/span> <span class=\"token string\">&#034;ERROR: missing <span class=\"token variable\">$f<\/span> (run from the repo root)&#034;<\/span> <span class=\"token operator\">&gt;<\/span><span class=\"token file-descriptor important\">&amp;2<\/span><br \/>\n        <span class=\"token builtin class-name\">exit<\/span> <span class=\"token number\">1<\/span><br \/>\n    <span class=\"token keyword\">fi<\/span><br \/>\n<span class=\"token keyword\">done<\/span><br \/>\n<span class=\"token keyword\">for<\/span> <span class=\"token for-or-select variable\">d<\/span> <span class=\"token keyword\">in<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$IMAGE_DIR<\/span>&#034;<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$MODEL_PATH<\/span>&#034;<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$VISION_TOWER<\/span>&#034;<\/span><span class=\"token punctuation\">;<\/span> <span class=\"token keyword\">do<\/span><br \/>\n    <span class=\"token keyword\">if<\/span> <span class=\"token punctuation\">[<\/span> <span class=\"token operator\">!<\/span> <span class=\"token parameter variable\">-d<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$d<\/span>&#034;<\/span> <span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">;<\/span> <span class=\"token keyword\">then<\/span><br \/>\n        <span class=\"token builtin class-name\">echo<\/span> <span class=\"token string\">&#034;ERROR: missing <span class=\"token variable\">$d<\/span> (run from the repo root)&#034;<\/span> <span class=\"token operator\">&gt;<\/span><span class=\"token file-descriptor important\">&amp;2<\/span><br \/>\n        <span class=\"token builtin class-name\">exit<\/span> <span class=\"token number\">1<\/span><br \/>\n    <span class=\"token keyword\">fi<\/span><br \/>\n<span class=\"token keyword\">done<\/span><br \/>\n<span class=\"token comment\"># 665k \u7684\u56fe\u7247\u6563\u5728 5 \u4e2a\u5b50\u76ee\u5f55\u91cc, \u5c11\u4efb\u4f55\u4e00\u4e2a\u90fd\u4f1a\u5728\u8bad\u7ec3\u4e2d\u9014\u624d\u70b8\u3002<\/span><br \/>\n<span class=\"token keyword\">for<\/span> <span class=\"token for-or-select variable\">sub<\/span> <span class=\"token keyword\">in<\/span> coco gqa ocr_vqa textvqa vg<span class=\"token punctuation\">;<\/span> <span class=\"token keyword\">do<\/span><br \/>\n    <span class=\"token keyword\">if<\/span> <span class=\"token punctuation\">[<\/span> <span class=\"token operator\">!<\/span> <span class=\"token parameter variable\">-d<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$IMAGE_DIR<\/span>\/<span class=\"token variable\">$sub<\/span>&#034;<\/span> <span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">;<\/span> <span class=\"token keyword\">then<\/span><br \/>\n        <span class=\"token builtin class-name\">echo<\/span> <span class=\"token string\">&#034;ERROR: missing <span class=\"token variable\">$IMAGE_DIR<\/span>\/<span class=\"token variable\">$sub<\/span>&#034;<\/span> <span class=\"token operator\">&gt;<\/span><span class=\"token file-descriptor important\">&amp;2<\/span><br \/>\n        <span class=\"token builtin class-name\">exit<\/span> <span class=\"token number\">1<\/span><br \/>\n    <span class=\"token keyword\">fi<\/span><br \/>\n<span class=\"token keyword\">done<\/span><br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token comment\"># \u5df2\u7ecf\u6709\u4eba\u8dd1\u8fc7\u5c31\u63d0\u9192\u4e00\u4e0b, \u514d\u5f97\u624b\u6ed1\u8986\u76d6\u3002<\/span><br \/>\n<span class=\"token keyword\">if<\/span> <span class=\"token punctuation\">[<\/span> <span class=\"token parameter variable\">-e<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$OUTPUT_DIR<\/span>&#034;<\/span> <span class=\"token punctuation\">]<\/span> <span class=\"token operator\">&amp;&amp;<\/span> <span class=\"token punctuation\">[<\/span> <span class=\"token parameter variable\">-n<\/span> <span class=\"token string\">&#034;<span class=\"token variable\"><span class=\"token variable\">$(<\/span><span class=\"token function\">ls<\/span> <span class=\"token parameter variable\">-A<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$OUTPUT_DIR<\/span>&#034;<\/span> <span class=\"token operator\"><span class=\"token file-descriptor important\">2<\/span>&gt;<\/span>\/dev\/null<span class=\"token variable\">)<\/span><\/span>&#034;<\/span> <span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">;<\/span> <span class=\"token keyword\">then<\/span><br \/>\n    <span class=\"token builtin class-name\">echo<\/span> <span class=\"token string\">&#034;WARNING: <span class=\"token variable\">$OUTPUT_DIR<\/span> already exists and is not empty.&#034;<\/span> <span class=\"token operator\">&gt;<\/span><span class=\"token file-descriptor important\">&amp;2<\/span><br \/>\n<span class=\"token keyword\">fi<\/span><br \/>\n<span class=\"token comment\">#<\/span><br \/>\n<span class=\"token builtin class-name\">echo<\/span> <span class=\"token string\">&#034;preflight OK&#034;<\/span><br \/>\n<span class=\"token comment\">#<\/span><br \/>\ndeepspeed llava\/train\/train_mem.py <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;deepspeed<\/span> .\/scripts\/zero3.json <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;model_name_or_path<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$MODEL_PATH<\/span>&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;version<\/span> v1 <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;data_path<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$DATA_JSON<\/span>&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;image_folder<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$IMAGE_DIR<\/span>&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;vision_tower<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$VISION_TOWER<\/span>&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;pretrain_mm_mlp_adapter<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$MM_PROJECTOR<\/span>&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;mm_projector_type<\/span> mlp2x_gelu <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;mm_vision_select_layer<\/span> <span class=\"token parameter variable\">-2<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;mm_use_im_start_end<\/span> False <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;mm_use_im_patch_token<\/span> False <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;image_aspect_ratio<\/span> pad <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;group_by_modality_length<\/span> True <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;bf16<\/span> True <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;output_dir<\/span> <span class=\"token string\">&#034;<span class=\"token variable\">$OUTPUT_DIR<\/span>&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;num_train_epochs<\/span> <span class=\"token number\">1<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;per_device_train_batch_size<\/span> <span class=\"token number\">16<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;per_device_eval_batch_size<\/span> <span class=\"token number\">4<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;gradient_accumulation_steps<\/span> <span class=\"token number\">1<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;evaluation_strategy<\/span> <span class=\"token string\">&#034;no&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;save_strategy<\/span> <span class=\"token string\">&#034;steps&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;save_steps<\/span> <span class=\"token number\">500<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;save_total_limit<\/span> <span class=\"token number\">2<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;learning_rate<\/span> 2e-5 <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;weight_decay<\/span> <span class=\"token number\">0<\/span>. <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;warmup_ratio<\/span> <span class=\"token number\">0.03<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;lr_scheduler_type<\/span> <span class=\"token string\">&#034;cosine&#034;<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;logging_steps<\/span> <span class=\"token number\">1<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;tf32<\/span> True <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;model_max_length<\/span> <span class=\"token number\">2048<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;gradient_checkpointing<\/span> True <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;dataloader_num_workers<\/span> <span class=\"token number\">4<\/span> <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;lazy_preprocess<\/span> True <span class=\"token punctuation\">\\\\<\/span><br \/>\n    <span class=\"token parameter variable\">&#8211;report_to<\/span> none<\/p>\n<h3>3. \u5b9e\u9a8c\u7ed3\u679c&#xff1a;POPE \u548c MME<\/h3>\n<p>\u2003\u2003\u8dd1\u5b8c\u540e&#xff0c;\u6211\u7528 POPE \u548c MME \u4e24\u4e2a\u57fa\u51c6\u505a\u4e86\u8bc4\u6d4b\u3002<br \/>\n\u2003\u2003POPE \u6d4b\u5e7b\u89c9&#xff0c;\u770b\u6a21\u578b\u4f1a\u4e0d\u4f1a\u778e\u8bf4\u6709\u3001\u778e\u8bf4\u6ca1\u6709\u3002<br \/>\n\u2003\u2003MME \u5206\u611f\u77e5\u548c\u8ba4\u77e5\u4e24\u5927\u7c7b&#xff0c;\u5341\u51e0\u4e2a\u5b50\u4efb\u52a1\u3002<br \/>\n\u2003\u2003\u7ed3\u679c\u548c\u8bba\u6587\u57fa\u672c\u5bf9\u9f50&#xff0c;\u811a\u672c\u6ca1\u628a\u6a21\u578b\u6539\u574f\u3002<br \/>\n\u2003\u2003\u6210\u7ee9\u5982\u4e0b\u3002<br \/>\n\u2003\u2003<br \/>\n<img decoding=\"async\" src=\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/10\/20261001082504-6abe18e069910.png\" alt=\"\u5916\u94fe\u56fe\u7247\u8f6c\u5b58\u5931\u8d25,\u6e90\u7ad9\u53ef\u80fd\u6709\u9632\u76d7\u94fe\u673a\u5236,\u5efa\u8bae\u5c06\u56fe\u7247\u4fdd\u5b58\u4e0b\u6765\u76f4\u63a5\u4e0a\u4f20\u3002\" \/><\/p>\n<hr \/>\n<h2>\u603b\u7ed3<\/h2>\n<p>\u2003\u2003LLaVA-1.5 \u7684\u7ed3\u6784\u5f88\u6734\u7d20&#xff1a;ViT \u770b\u56fe&#xff0c;MLP \u5bf9\u9f50&#xff0c;LLM \u8bf4\u8bdd\u3002<br \/>\n\u2003\u2003\u590d\u73b0\u7684\u96be\u70b9\u4e0d\u5728\u6a21\u578b&#xff0c;\u5728\u5de5\u7a0b\u7ec6\u8282\u3002<br \/>\n\u2003\u2003\u5148\u6309\u5c0f\u95f4\u9694\u4fdd\u5b58&#xff0c;\u518d\u8c08\u8dd1\u591a\u4e45\u3002\u5d29\u4e00\u6b21\u4e22 4 \u5c0f\u65f6&#xff0c;\u6bd4\u591a\u82b1 1 \u5c0f\u65f6\u5199\u76d8\u4e8f\u5f97\u591a\u3002<br \/>\n\u2003\u2003\u57fa\u672c\u590d\u73b0\u5df2\u5b8c\u6210&#xff0c;\u540e\u7eed\u4f1a\u51fa\u6e90\u7801\u89e3\u8bfb\u3002\u60f3\u8981\u8bad\u7ec3\u65e5\u5fd7\u7684\u540c\u5b66\u53ef\u4ee5\u5173\u6ce8\u52a0\u79c1\u4fe1\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u6587\u7ae0\u76ee\u5f55\u524d\u8a00\u4e00\u3001\u539f\u7406&#xff1a;\u4e09\u5757\u79ef\u6728\u62fc\u51fa\u6765\u7684\u591a\u6a21\u60011. \u7ed3\u6784&#xff1a;LLM  Align Layer  ViT2. \u635f\u5931\u8ba1\u7b97\u4e8c\u3001\u8bba\u6587\u91cc\u7684\u5b9e\u9a8c\u4e09\u3001\u590d\u73b0&#xff1a;8\u00d7L20 \u4e0a\u7684\u53cc\u9636\u6bb5\u8bad\u7ec31. stage1&#xff1a;Align \u5c42\u5bf9\u9f50\u8bad\u7ec32. stage2&#xff1a;Visual \u6307\u4ee4\u5fae\u8c033. \u5b9e\u9a8c\u7ed3\u679c&#xff1a;POPE \u548c MME\u603b\u7ed3\u524d\u8a00LLaVA \u662f\u5f00\u6e90\u591a\u6a21\u6001\u5927\u6a21\u578b\u91cc\u6700\u597d\u4e0a\u624b\u7684\u9879\u76ee\u4e4b\u4e00\u3002\u5b83\u628a\u89c6\u89c9\u7f16\u7801\u5668\u548c\u8bed\u8a00\u6a21\u578b\u62fc\u5728\u4e00\u8d77&#xff0c;\u5206\u4e24<\/p>\n","protected":false},"author":2,"featured_media":111697,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[11653,50],"topic":[],"class_list":["post-111701","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-server","tag-11653","tag-50"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v20.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>LLaVA1.5-7B\u590d\u73b0 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.wsisp.com\/helps\/111701.html\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"LLaVA1.5-7B\u590d\u73b0 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"og:description\" content=\"\u6587\u7ae0\u76ee\u5f55\u524d\u8a00\u4e00\u3001\u539f\u7406&#xff1a;\u4e09\u5757\u79ef\u6728\u62fc\u51fa\u6765\u7684\u591a\u6a21\u60011. \u7ed3\u6784&#xff1a;LLM Align Layer ViT2. \u635f\u5931\u8ba1\u7b97\u4e8c\u3001\u8bba\u6587\u91cc\u7684\u5b9e\u9a8c\u4e09\u3001\u590d\u73b0&#xff1a;8\u00d7L20 \u4e0a\u7684\u53cc\u9636\u6bb5\u8bad\u7ec31. stage1&#xff1a;Align \u5c42\u5bf9\u9f50\u8bad\u7ec32. stage2&#xff1a;Visual \u6307\u4ee4\u5fae\u8c033. \u5b9e\u9a8c\u7ed3\u679c&#xff1a;POPE \u548c MME\u603b\u7ed3\u524d\u8a00LLaVA \u662f\u5f00\u6e90\u591a\u6a21\u6001\u5927\u6a21\u578b\u91cc\u6700\u597d\u4e0a\u624b\u7684\u9879\u76ee\u4e4b\u4e00\u3002\u5b83\u628a\u89c6\u89c9\u7f16\u7801\u5668\u548c\u8bed\u8a00\u6a21\u578b\u62fc\u5728\u4e00\u8d77&#xff0c;\u5206\u4e24\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.wsisp.com\/helps\/111701.html\" \/>\n<meta property=\"og:site_name\" content=\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"article:published_time\" content=\"2026-10-01T08:25:05+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/10\/20261001082503-6abe18dfd14fc.png\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/111701.html\",\"url\":\"https:\/\/www.wsisp.com\/helps\/111701.html\",\"name\":\"LLaVA1.5-7B\u590d\u73b0 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"isPartOf\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\"},\"datePublished\":\"2026-10-01T08:25:05+00:00\",\"dateModified\":\"2026-10-01T08:25:05+00:00\",\"author\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/111701.html#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.wsisp.com\/helps\/111701.html\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/111701.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.wsisp.com\/helps\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"LLaVA1.5-7B\u590d\u73b0\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\",\"url\":\"https:\/\/www.wsisp.com\/helps\/\",\"name\":\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"description\":\"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"contentUrl\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"caption\":\"admin\"},\"sameAs\":[\"http:\/\/wp.wsisp.com\"],\"url\":\"https:\/\/www.wsisp.com\/helps\/author\/admin\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"LLaVA1.5-7B\u590d\u73b0 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.wsisp.com\/helps\/111701.html","og_locale":"zh_CN","og_type":"article","og_title":"LLaVA1.5-7B\u590d\u73b0 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","og_description":"\u6587\u7ae0\u76ee\u5f55\u524d\u8a00\u4e00\u3001\u539f\u7406&#xff1a;\u4e09\u5757\u79ef\u6728\u62fc\u51fa\u6765\u7684\u591a\u6a21\u60011. \u7ed3\u6784&#xff1a;LLM Align Layer ViT2. \u635f\u5931\u8ba1\u7b97\u4e8c\u3001\u8bba\u6587\u91cc\u7684\u5b9e\u9a8c\u4e09\u3001\u590d\u73b0&#xff1a;8\u00d7L20 \u4e0a\u7684\u53cc\u9636\u6bb5\u8bad\u7ec31. stage1&#xff1a;Align \u5c42\u5bf9\u9f50\u8bad\u7ec32. stage2&#xff1a;Visual \u6307\u4ee4\u5fae\u8c033. \u5b9e\u9a8c\u7ed3\u679c&#xff1a;POPE \u548c MME\u603b\u7ed3\u524d\u8a00LLaVA \u662f\u5f00\u6e90\u591a\u6a21\u6001\u5927\u6a21\u578b\u91cc\u6700\u597d\u4e0a\u624b\u7684\u9879\u76ee\u4e4b\u4e00\u3002\u5b83\u628a\u89c6\u89c9\u7f16\u7801\u5668\u548c\u8bed\u8a00\u6a21\u578b\u62fc\u5728\u4e00\u8d77&#xff0c;\u5206\u4e24","og_url":"https:\/\/www.wsisp.com\/helps\/111701.html","og_site_name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","article_published_time":"2026-10-01T08:25:05+00:00","og_image":[{"url":"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/10\/20261001082503-6abe18dfd14fc.png"}],"author":"admin","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"admin","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"6 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.wsisp.com\/helps\/111701.html","url":"https:\/\/www.wsisp.com\/helps\/111701.html","name":"LLaVA1.5-7B\u590d\u73b0 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","isPartOf":{"@id":"https:\/\/www.wsisp.com\/helps\/#website"},"datePublished":"2026-10-01T08:25:05+00:00","dateModified":"2026-10-01T08:25:05+00:00","author":{"@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41"},"breadcrumb":{"@id":"https:\/\/www.wsisp.com\/helps\/111701.html#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.wsisp.com\/helps\/111701.html"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.wsisp.com\/helps\/111701.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.wsisp.com\/helps"},{"@type":"ListItem","position":2,"name":"LLaVA1.5-7B\u590d\u73b0"}]},{"@type":"WebSite","@id":"https:\/\/www.wsisp.com\/helps\/#website","url":"https:\/\/www.wsisp.com\/helps\/","name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","description":"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41","name":"admin","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/","url":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","contentUrl":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","caption":"admin"},"sameAs":["http:\/\/wp.wsisp.com"],"url":"https:\/\/www.wsisp.com\/helps\/author\/admin"}]}},"_links":{"self":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/111701","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/comments?post=111701"}],"version-history":[{"count":0,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/111701\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media\/111697"}],"wp:attachment":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media?parent=111701"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/categories?post=111701"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/tags?post=111701"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/topic?post=111701"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}