Changhan Wang

About Me

I am a Research Engineer at Facebook AI Research (FAIR), mainly working on natural language and speech processing. Prior to Facebook, I was a Software Engineer at Tapad. I earned my Master's degree in Computer Science from University of Michigan, Ann Arbor, where I worked with Prof. Honglak Lee on analyzing medical images with deep learning. I hold a Bachelor's degree (Hons) in Mathematics from Zhejiang University.

What's New?

2021-08-26: One demo paper was accepted to EMNLP 2021.
2021-06-18: One paper was accepted to IWSLT 2021. Our speech translation system won the 1st place in the multilingual translation (unconstrained) track.
2021-06-02: Two papers (1, 2) were accepted to INTERSPEECH 2021.
2021-05-05: Four papers (1, 2, 3, 4) were accepted to ACL 2021.
2020-09-30: One paper was accepted to COLING 2020.

2020-09-30

paper

WMT 2020

1st place

2nd place

2020-09-15: One demo paper was accepted to EMNLP 2020.
2020-09-10: One demo paper was accepted to AACL 2020.
2020-07-24: Two papers (1,2) were accepted to INTERSPEECH 2020.
2020-02-11: One paper was accepted to LREC 2020. Data & code is available.
2019-11-10: One paper was accepted to AAAI 2020. Code is available.
2019-11-03: VizSeq (an analysis toolkit for natural language generation) released. Check out our blog.
2019-09-03: One paper was accepted to NeurIPS 2019. Code is available.
2019-08-12: One demo paper was accepted to EMNLP 2019. Code is available.
2019-05-06: One paper was accepted to CVPR 2019 FATE/CV & CV4GC workshops. Check out our blog.
2018-08-10: One paper was accepted to EMNLP 2018. Code is available.
2018-05-04: One paper was accepted to ACL 2018 CALCS workshop. Our system won the 1st place in the MS Arabic-Egyptian code-switched Named Entity Recognition shared task.

Service

Program Committee/Reviewer: AAAI, ACL, EMNLP, NAACL, IWSLT, PLOS Computational Biology

Open Source

VizSeq
An analysis toolkit for natural language generation. Blog
ClassyVision
An end-to-end PyTorch framework for image and video classification.

Publications

VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation

Abstract: We introduce VoxPopuli, a large-scale multilingual corpus providing 100K hours of unlabelled speech data in 23 languages. It is the largest open data to date for unsupervised representation learning as well as semi-supervised learning. VoxPopuli also contains 1.8K hours of transcribed speeches in 16 languages and their aligned oral interpretations into 5 other languages totaling 5.1K hours. We provide speech recognition baselines and validate the versatility of VoxPopuli unlabelled data in semi-supervised learning under challenging out-of-domain settings. We will release the corpus at https://github.com/facebookresearch/voxpopuli under an open license.

Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, Emmanuel Dupoux
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP), 2021
Multilingual Speech Translation with Efficient Finetuning of Pretrained Models

Abstract: We present a simple yet effective approach to build multilingual speech-to-text (ST) translation by efficient transfer learning from pretrained speech encoder and text decoder. Our key finding is that a minimalistic LNA (LayerNorm and Attention) finetuning can achieve zero-shot crosslingual and cross-modality transfer ability by only finetuning less than 10% of the pretrained parameters. This enables effectively leveraging large pretrained models with low training cost. Using wav2vec 2.0 for acoustic modeling, and mBART for multilingual text generation, our approach advanced the new state-of-the-art for 34 translation directions (and surpassing cascaded ST for 23 of them) on large-scale multilingual ST benchmark CoVoST 2 (+6.4 BLEU on average across 15 En-X directions and +5.1 BLEU on average across 19 X-En directions). Our approach demonstrates strong zero-shot performance in a many-to-many multilingual model (+5.7 BLEU on average across 18 non-English directions), making it an appealing approach for attaining high-quality speech translation with improved parameter and data efficiency.

Xian Li*, Changhan Wang*, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, Michael Auli (* equal contribution)
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP), 2021
Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation

Abstract: We introduce dual-decoder Transformer, a new model architecture that jointly performs automatic speech recognition (ASR) and multilingual speech translation (ST). Our models are based on the original Transformer architecture (Vaswani et al., 2017) but consist of two decoders, each responsible for one task (ASR or ST). Our major contribution lies in how these decoders interact with each other: one decoder can attend to different information sources from the other via a dual-attention mechanism. We propose two variants of these architectures corresponding to two different levels of dependencies between the decoders, called the parallel and cross dual-decoder Transformers, respectively. Extensive experiments on the MuST-C dataset show that our models outperform the previously-reported highest translation performance in the multilingual settings, and outperform as well bilingual one-to-one results. Furthermore, our parallel models demonstrate no trade-off between ASR and ST compared to the vanilla multi-task architecture.

Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, Laurent Besacier
Proceedings of the 28th International Conference on Computational Linguistics (COLING), 2020
fairseq S2T: Fast Speech-to-Text Modeling with fairseq

Abstract: We introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation. It follows fairseq's careful design for scalability and extensibility. We provide end-to-end workflows from data pre-processing, model training to offline (online) inference. We implement state-of-the-art RNN-based as well as Transformer-based models and open-source detailed training recipes. Fairseq's machine translation models and language models can be seamlessly integrated into S2T workflows for multi-task learning or transfer learning. Fairseq S2T documentation and examples are available at https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text.

Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, Juan Pino
Proceedings of the 2020 Conference of the Asian Chapter of the Association for Computational Linguistics (AACL): System Demonstrations, 2020
Facebook AI's WMT20 News Translation Task Submission

Abstract: This paper describes Facebook AI’s submission to WMT20 shared news translation task. We focus on the low resource setting and participate in two language pairs, Tamil <-> English and Inuktitut <-> English, where there are limited out-of-domain bitext and monolingual data. We approach the low resource problem using two main strategies, leveraging all available data and adapting the system to the target news domain. We explore techniques that leverage bitext and monolingual data from all languages, such as self-supervised model pretraining, multilingual models, data augmentation, and reranking. To better adapt the translation system to the test domain, we explore dataset tagging and fine-tuning on in-domain data. We observe that different techniques provide varied improvements based on the available data of the language pair. Based on the finding, we integrate these techniques into one training pipeline. For En->Ta, we explore an unconstrained setup with additional Tamil bitext and monolingual data and show that further improvement can be obtained. On the test set, our best submitted systems achieve 21.5 and 13.7 BLEU for Ta->En and En->Ta respectively, and 27.9 and 13.0 for Iu->En and En->Iu respectively.

Peng-Jen Chen, Ann Lee, Changhan Wang, Naman Goyal, Angela Fan, Mary Williamson, Jiatao Gu
Proceedings of the Fifth Conference on Machine Translation (WMT), 2020
SimulEval: An Evaluation Toolkit for Simultaneous Translation

Abstract: Simultaneous translation on both text and speech focuses on a real-time and low-latency scenario where the model starts translating before reading the complete source input. Evaluating simultaneous translation models is more complex than offline models because the latency is another factor to consider in addition to translation quality. The research community, despite its growing focus on novel modeling approaches to simultaneous translation, currently lacks a universal evaluation procedure. Therefore, we present SimulEval, an easy-to-use and general evaluation toolkit for both simultaneous text and speech translation. A server-client scheme is introduced to create a simultaneous translation scenario, where the server sends source input and receives predictions for evaluation and the client executes customized policies. Given a policy, it automatically performs simultaneous decoding and collectively reports several popular latency metrics. We also adapt latency metrics from text simultaneous translation to the speech task. Additionally, SimulEval is equipped with a visualization interface to provide better understanding of the simultaneous decoding process of a system. SimulEval has already been extensively used for the IWSLT 2020 shared task on simultaneous speech translation. Code will be released upon publication.

Xutai Ma, Mohammad Javad Dousti, Changhan Wang, Jiatao Gu, Juan Pino
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, 2020
Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation

Abstract: Transfer learning from high-resource languages is known to be an efficient way to improve end-to-end automatic speech recognition (ASR) for low-resource languages. Pre-trained or jointly trained encoder-decoder models, however, do not share the language modeling (decoder) for the same language, which is likely to be inefficient for distant target languages. We introduce speech-to-text translation (ST) as an auxiliary task to incorporate additional knowledge of the target language and enable transferring from that target language. Specifically, we first translate high-resource ASR transcripts into a target low-resource language, with which a ST model is trained. Both ST and target ASR share the same attention-based encoder-decoder architecture and vocabulary. The former task then provides a fully pre-trained model for the latter, bringing up to 24.6% word error rate (WER) reduction to the baseline (direct transfer from high-resource ASR). We show that training ST with human translations is not necessary. ST trained with machine translation (MT) pseudo-labels brings consistent gains. It can even outperform those using human labels when transferred to target ASR by leveraging only 500K MT examples. Even with pseudo-labels from low-resource MT (200K examples), ST-enhanced transfer brings up to 8.9% WER reduction to direct transfer.

Changhan Wang, Juan Pino, Jiatao Gu
The 21st Annual Conference of the International Speech Communication Association (INTERSPEECH), 2020
Self-Supervised Representations Improve End-to-End Speech Translation

Abstract: End-to-end speech-to-text translation can provide a simpler and smaller system but is facing the challenge of data scarcity. Pre-training methods can leverage unlabeled data and have been shown to be effective on data-scarce settings. In this work, we explore whether self-supervised pre-trained speech representations can benefit the speech translation task in both high-and low-resource settings, whether they can transfer well to other languages, and whether they can be effectively combined with other common methods that help improve low-resource end-to-end speech translation such as using a pre-trained high-resource speech recognition system. We demonstrate that self-supervised pre-trained features can consistently improve the translation performance, and cross-lingual transfer allows to extend to a variety of languages without or with little tuning.

Anne Wu, Changhan Wang, Juan Pino, Jiatao Gu
The 21st Annual Conference of the International Speech Communication Association (INTERSPEECH), 2020
Findings of the IWSLT 2020 Evaluation Campaign

Abstract: The evaluation campaign of the International Conference on Spoken Language Translation (IWSLT 2020) featured this year six challenge tracks:(i) Simultaneous speech translation,(ii) Video speech translation,(iii) Offline speech translation,(iv) Conversational speech translation,(v) Open domain translation, and (vi) Non-native speech translation. A total of teams participated in at least one of the tracks. This paper introduces each track’s goal, data and evaluation metrics, and reports the results of the received submissions.

Ebrahim Ansari, Nguyen Bach, Ondřej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, Fei Huang, Kevin Knight, Xutai Ma, Ajay Nagesh, Matteo Negri, Jan Niehues, Juan Pino, Elizabeth Salesky, Xing Shi, Sebastian Stüker, Marco Turchi, Alex Waibel, Changhan Wang
Proceedings of the 17th International Conference on Spoken Language Translation, 2020
CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus

Abstract: Spoken language translation has recently witnessed a resurgence in popularity, thanks to the development of end-to-end models and the creation of new corpora, such as Augmented LibriSpeech and MuST-C. Existing datasets involve language pairs with English as a source language, involve very specific domains or are low resource. We introduce CoVoST, a multilingual speech-to-text translation corpus from 11 languages into English, diversified with over 11,000 speakers and over 60 accents. We describe the dataset creation methodology and provide empirical evidence of the quality of the data. We also provide initial benchmarks, including, to our knowledge, the first end-to-end many-to-one multilingual models for spoken language translation. CoVoST is released under CC0 license and free to use. We also provide additional evaluation data derived from Tatoeba under CC licenses.

Changhan Wang, Juan Pino, Anne Wu, Jiatao Gu
Proceedings of the Twelfth International Conference on Language Resources and Evaluation (LREC), 2020
Neural Machine Translation with Byte-Level Subwords

Abstract: Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can unnecessarily take up vocabulary slots and limit its compactness. Representing text at the level of bytes and using the 256 byte set as vocabulary is a potential solution to this issue. High computational cost has however prevented it from being widely deployed or used in practice. In this paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE), which is compacter than character vocabulary and has no out-of-vocabulary tokens, but is more efficient than using pure bytes only is. We claim that contextualizing BBPE embeddings is necessary, which can be implemented by a convolutional or recurrent layer. Our experiments show that BBPE has comparable performance to BPE while its size is only 1/8 of that for BPE. In the multilingual setting, BBPE maximizes vocabulary sharing across many languages and achieves better translation quality. Moreover, we show that BBPE enables transferring models between languages with non-overlapping character sets.

Changhan Wang, Kyunghyun Cho, Jiatao Gu
The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI), 2020
VizSeq: A Visual Analysis Toolkit for Text Generation Tasks

Abstract: Automatic evaluation of text generation tasks (e.g. machine translation, text summarization, image captioning and video description) usually relies heavily on task-specific metrics, such as BLEU and ROUGE. They, however, are abstract numbers and are not perfectly aligned with human assessment. This suggests inspecting detailed examples as a complement to identify system error patterns. In this paper, we present VizSeq, a visual analysis toolkit for instance-level and corpus-level system evaluation on a wide variety of text generation tasks. It supports multimodal sources and multiple text references, providing visualization in Jupyter notebook or a web app interface. It can be used locally or deployed onto public servers for centralized data hosting and benchmarking. It covers most common n-gram based metrics accelerated with multiprocessing, and also provides latest embedding-based metrics such as BERTScore.

Changhan Wang, Anirudh Jain, Danlu Chen, Jiatao Gu
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, 2019
Levenshtein Transformer

Abstract: Modern neural sequence generation models are built to either generate tokens step-by-step from scratch or (iteratively) modify a sequence of tokens bounded by a fixed length. In this work, we develop Levenshtein Transformer, a new partially autoregressive model devised for more flexible and amenable sequence generation. Unlike previous approaches, the atomic operations of our model are insertion and deletion. The combination of them facilitates not only generation but also sequence refinement allowing dynamic length changes. We also propose a set of new training techniques dedicated at them, effectively exploiting one as the other's learning signal thanks to their complementary nature. Experiments applying the proposed model achieve comparable performance but much-improved efficiency on both generation (e.g. machine translation, text summarization) and refinement tasks (e.g. automatic post-editing). We further confirm the flexibility of our model by showing a Levenshtein Transformer trained by machine translation can straightforwardly be used for automatic post-editing.

Jiatao Gu, Changhan Wang, Jake Zhao
Advances in Neural Information Processing Systems (NeurIPS), 2019
Does Object Recognition Work for Everyone?

Abstract: The paper analyzes the accuracy of publicly available object-recognition systems on a geographically diverse dataset. This dataset contains household items and was designed to have a more representative geographical coverage than commonly used image datasets in object recognition. We find that the systems perform relatively poorly on household items that commonly occur in countries with a low household income. Qualitative analyses suggest the drop in performance is primarily due to appearance differences within an object class (eg, dish soap) and due to items appearing in a different context (eg, toothbrushes appearing outside of bathrooms). The results of our study suggest that further work is needed to make object-recognition systems work equally well for people across different countries and income levels.

Terrance DeVries*, Ishan Misra*, Changhan Wang*, Laurens van der Maaten (* equal contribution)
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019
Dynamic Meta-Embeddings for Improved Sentence Representations

Abstract: While one of the first steps in many NLP systems is selecting what pre-trained word embeddings to use, we argue that such a step is better left for neural networks to figure out by themselves. To that end, we introduce dynamic meta-embeddings, a simple yet effective method for the supervised learning of embedding ensembles, which leads to state-of-the-art performance within the same model class on a variety of tasks. We subsequently show how the technique can be used to shed new light on the usage of word embeddings in NLP systems.

Douwe Kiela, Changhan Wang, Kyunghyun Cho
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018
Code-Switched Named Entity Recognition with Embedding Attention

Abstract: We describe our work for the CALCS 2018 shared task on named entity recognition on code-switched data. Our system ranked first place for MS Arabic-Egyptian named entity recognition and third place for English-Spanish.

Changhan Wang, Kyunghyun Cho, Douwe Kiela
Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, 2018
A Unified Framework for Automatic Wound Segmentation and Analysis with Deep Convolutional Neural Networks

Abstract: Wound surface area changes over multiple weeks are highly predictive of the wound healing process. Furthermore, the quality and quantity of the tissue in the wound bed also offer important prognostic information. Unfortunately, accurate measurements of wound surface area changes are out of reach in the busy wound practice setting. Currently, clinicians estimate wound size by estimating wound width and length using a scalpel after wound treatment, which is highly inaccurate. To address this problem, we propose an integrated system to automatically segment wound regions and analyze wound conditions in wound images. Different from previous segmentation techniques which rely on hand-crafted features or unsupervised approaches, our proposed deep learning method jointly learns task-relevant visual features and performs wound segmentation. Moreover, learned features are applied to further analysis of wounds in two ways: infection detection and healing progress prediction. To the best of our knowledge, this is the first attempt to automate long-term predictions of general wound healing progress. Our method is computationally efficient and takes less than 5 seconds per wound image (480 by 640 pixels) on a typical laptop computer. Our evaluations on a large-scale wound database demonstrate the effectiveness and reliability of the proposed system.

Changhan Wang, Xinchen Yan, Max Smith, Kanika Kochhar, Marcie Rubin, Stephen M Warren, James Wrobel, Honglak Lee
The 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2015
InSide: Interactive Sketching for Image Database Exploration

Abstract: We propose an interactive sketching tool for exploring image database, called InSide. Our main contribution is a new solution of interactive image exploration that dynamically adapts to users' sketching and provides mixed feedback. A position-aware matching approach is proposed for InSide in order to support translation-free sketch searching. Based on demonstrated results, our method outperforms state-of-the-art approaches in aspects of user interface and matching results.

Hongxin Zhang, Dongyu Liu, Changhan Wang
International Conference on Computer-Aided Design and Computer Graphics (CAD/Graphics), 2013

Changhan Wang（王昌翰）

About Me

What's New?

Service

Open Source

Publications

Changhan Wang
（王昌翰）