Thinkwee's Blog

Some very-personal questions, assumptions and predictions on the future after the large model era. I hope to keep it a habit for writing such future-ask post for every half year to keep me thinking about the “next token” in the AI era. This post is about Compression, World Model, Agent and Alignment.

A simple note on the RL used in single-agent and multi-agent.

• A webpage based on ChatGPT-Shortcut that shows some interesting debates that took place between GPTs.
• The experience website is here

• Record of recent task reconstruction methods based on templates, a particularly interesting direction since the appearance of GPT-3. These methods typically design prompts for tasks, converting samples and tasks into natural language templates, which are then directly input into pre-trained language models to generate text, thereby indirectly completing the tasks. The construction of prompts standardizes the form of downstream tasks and pre-trained tasks (language models), achieving good results in few-shot learning. Key papers to read include the following nine:
  • Early work that converts questions into natural language and uses pre-trained language models for answers:
    • (Harvard) Commonsense Knowledge Mining from Pretrained Models
    • (Heidelberg) Argumentative Relation Classification as Plausibility Ranking
    • (NVIDIA) Zero-shot Text Classification With Generative Language Models
  • The PET approach, Pattern Exploiting Training:
    • (LMU) Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference
    • (LMU) It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
    • (UNC) Improving and Simplifying Pattern Exploiting Training
  • Automatically constructing prompts, Automatically Searching Prompts:
    • (UCI, UCB) AUTOPROMPT: Eliciting Knowledge from Language Models with Automatically Generated Prompts
    • (Princeton, MIT) Making Pre-trained Language Models Better Few-shot Learners
    • (THU) GPT Understands, Too
      Read more »

• Record the methods of editing seq2seq in recent years, which have the advantages of high efficiency (partially autoregressive or non-autoregressive decoding) and less data hungry (small output vocabulary) for tasks with the same language input and output and minor changes (error correction, simplification, summarization).
• Mainly read five papers, sorted by their publication date on arXiv:
  • (LevT, Facebook) Levenshtein Transformer
  • (Huawei) EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing
  • (LaserTagger, Google) Encode, Tag, Realize: High-Precision Text Editing
  • (PIE) Parallel Iterative Edit Models for Local Sequence Transduction
  • (Google) Felix: Flexible Text Editing Through Tagging and Insertion

A brief review of the VC dimension. All discussions are based on the simple case of binary classification.

Read Dr. Bang Liu’s paper Natural Language Processing and Text Mining with Graph-Structured Representations from the University of Alberta and take some notes.

Study notes for Stanford CS224W: Machine Learning with Graphs by Jure Leskovec.

A brief note on the CLSciSumm Workshop that the CIST lab participated in, the main focus is on methods. The experiments are analysised in detail in papers.
Papers:

• 2016
• 2017
• 2018
• 2019

Record the incremental decoding processing of parallel decoding models such as CNN seq2seq and Transformer in the inference phase in Fairseq.

note for A Primer in BERTology: What we know about how BERT works

reading note for STRUCTURED NEURAL SUMMARIZATION.

Long time no see, SVM.

Reading note for reformer.

Paper reading on

• GNN Pooling
• Discourse-Aware Summarization
• Siamese BERT
• Large Chatbot

Note for Hierarchical Latent Dirichlet Allocation

• Knowledge Graph Special Collection
  • Entity Alignment in Cross-lingual Knowledge Graphs
  • Knowledge Graph Language Model
  • Dynamic Knowledge Graph Dialogue Generation
  • Graph2Seq
  • Graph Matching Network
  • Dynamic Knowledge Graph Update
  • Attention-based Embeddings for Relation Prediction

Record some recent processing of heterogeneous information networks

• PathSim
• HGNN
• HGAN
• HGAN for text classification
• Attribute, Attributed Multiplex Heterogeneous Network
• Meta-graph Guided Random Walks

Graph-based Automatic Summary Related Paper Selection Reading

• AMR Generative Summary
• AMR Multi-document Summarization Two Papers
• pagerank in encoder attention
• Build a graph based on thematic modeling, use ILP for extractive summarization
• Multi-document Extractive Summary Based on GCN
• STRUCTURED NEURAL SUMMARIZATION

rl study note, minimalist style

• Q-learning
• Sarsa
• Sarsa($\lambda$)
• DQN
• Double DQN
• DQN with Prioritized Experience replay
• Dueling DQN
• Policy Gradient

Selected Reading of ACL/NAACL 2019 Automatic Summarization Papers

• DPPs Similarity Measurement Improvement

• STRASS: Backpropagation for Extractive Summarization

• Translate first, then generate the abstract

• Reading Comprehension + Automatic Abstract

• BiSET: Retrieve + Fast Rerank + Selective Encoding + Template Based

Note for paper “Cognitive Graph for Multi-Hop Reading Comprehension at Scale.”

Note for CorEx(Correlation Explaination).

Selected readings from ACL 2019 award-winning papers.

• Using Oracle for sentence-level teacher forcing
• speaker commitment
• A set of evaluation index frameworks applicable to abstracts, combining multiple indicators
• Zero-Shot Entity Linking

• Variational Autoencoder Learning Notes

• Reference Article:

  • Auto-Encoding Variational Bayes
  • Daniel Daza, The Variational Autoencoder
  • Sūshén’s VAE series
• On VAE, the original paper and the two blogs above have already explained it very clearly. I am just repeating and paraphrasing, just to go through it myself. If anyone reads this blog, I recommend reading these three reference sources first

• Record the mathematical derivation of GloVe word vectors, as the original paper does not derive the model graphically but rather calculates the objective function through pure mathematical operations. This design approach is very interesting, and it also writes out and compares the mathematical essence of word2vec.
• GloVe: Global Vectors for Word Representation

• Convolutional Sequence to Sequence

• Robust Unsupervised Cross-Lingual Word Embedding Mapping

Course Notes on Computational Linguistics, Reference Textbook: Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition.

Note for John Mount’s “The Equivalence of Logistic Regression and Maximum Entropy Models” and explains that this proof is a special case of the general derivation proof of the maximum entropy model introduced in statistical learning methods

Conclusion

• Maximum entropy model is softmax classification
• Under the balanced conditions of the general linear model, the model mapping function that satisfies the maximum entropy condition is the softmax function
• In the book on Statistical Machine Learning methods, a maximum entropy model defined under the feature function is presented, which, along with softmax regression, belongs to the class of log-linear models
• When the feature function extends from a binary function to the feature value itself, the maximum entropy model becomes a softmax regression model
• The maximum entropy maximizes conditional entropy, not the entropy of conditional probabilities, nor the entropy of joint probabilities.

Attended a light salon at Tsinghua University’s FIT, which introduced some advancements in machine reading comprehension. Interestingly, the PhD who spoke at 9 am also mentioned an unpublished work: BERT, which is very impressive and well-funded; it took eight p100 GPUs to train for a year. By 10:30, Machine Intelligence had already published a report, and by the afternoon, Zhihu was buzzing with discussions, saying that a new era for NLP had arrived… This salon is part of a series, and there may be future sessions on machine translation, deep Bayesian, transfer learning, and knowledge graphs, so if you have the time, you might as well listen and take notes.