==============================================
Content of MSR-IME-Feature corpus, version 1.0
Contact: Jianfeng Gao, jfgao@microsoft.com
July 2007
==============================================

---------------------------------------------------
Data format for feature files
---------------------------------------------------
tr.20b.80k.txt   --> training set, containing 79999 20-best samples (sentences)
te.100b.80k.txt  --> test set, containing 4999 100-best samples (sentences)
dev.100b.80k.txt --> dev set, containing 4999 100-best samples (sentences)

Each file is of the following format:

<Data>       --> <Sentence>*
<Sentence>   --> <Conversion>* [##sample##]
<Conversion> --> <BaseFeatureScore> <NE> <FID>*

<Data> contains a list of sentences, separated by [##sample##]
Each <Sentence> consists of a list of candidate <Conversion>s.
Each <Conversion> consists a <BaseFeatureScore>, which is the log probability of a word trigram model, <NE>, which is the number of errors (measured via editing distance) in this conversion, and a set of <FID>s.
<FID> is the ID of a feature that occurs in the conversion. The feature sets contain all word uni-gram and bi-gram features.

---------------------------------------------------
Results
---------------------------------------------------

See Section 3.2 in "Jianfeng Gao, Galen Andrew, Mark Johnson and Kristina Toutanova. 2007. A comparative study of parameter estimation methods for statistical natural language processing. In ACL 2007".