BERT：文字填空 vs. ChatGPT：文字接龙

Finetuning - 专才

使用方式：对预训练模型做改造

• Head
• Finetuning：预训练模型的参数作为初始参数，基于新的训练数据，更新模型参数
• Adapter：又称efficient finetuning，只需对adapter的参数做finetuning
  • Advantages
  • Examples: see adapterhub
    • Bitfit: only finetune bias
    • Houlsby: add a feed-forward layer
    • Prefix-tuning & LoRA: revise attention

Prompting - 通才

History（“所有问题都能以问答题的方式解决”）

• 2015 - Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• 2018 - The Natural Language Decathlon: Multitask Learning as Question Answering

使用方式

• In-context learning：范例
  • Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
    • Experiment
      • random labels：表现变差，但不显著
      • irrelevant input：表现显著变差
    • Conclusion：本来就会做情感分析，只是需要被指出需要做情感分析的任务（范例数量增加没有显著的效果提升）
  • arxiv.org/abs/2303.03…
    • The more flipped labels, the lower the accuracy of the model is, especially for larger models.
    • Classification: better performance than random guessing - even match the perforance of SVM in some cases
  • Learning in-context learning
    • 大模型无需finetuning即具有in-context learning的能力，但经过训练后表现会更好
• Instruction learning：题目叙述
  • Instruction-tuning - necessary (different from in-context learning)
    • 收集大量自然语言处理的任务与对应的数据集，进行finetuning
    • Examples
      • T0
      • FLAN
• Chain of Thought (CoT) Prompting
  • Few-shot-CoT & Zero-shot-CoT
  • Self-consistency
    • CoT prompting时，答案的diversity比较大，self-consistency更有意义
  • Least-to-most prompting
    • 难点：stage 1，对问题做简化 - in-context learning
• 用机器生成prompt
  • soft prompt
  • reinforcment learning
  • using an LM to find prompt

Comparison

• 专才的好处：专才在单一任务上有机会赢过通才
  • Is ChatGPT A Good Translator? A Preliminary Study
  • How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• 通才的好处：只要重新设计prompt就可以快速开发新功能，不需重新编程

Further references

AACL 2022 Tutorial: Recent Advances in Pre-trained Language Models: Why Do They Work and How to Use Them