1929 or 1989?	1929年还是1989年?
PARIS – As the economic crisis deepens and widens, the world has been searching for historical analogies to help us understand what has been happening.	巴黎-随着经济危机不断加深和蔓延，整个世界一直在寻找历史上的类似事件希望有助于我们了解目前正在发生的情况。
At the start of the crisis, many people likened it to 1982 or 1973, which was reassuring, because both dates refer to classical cyclical downturns.	一开始，很多人把这次危机比作1982年或1973年所发生的情况，这样得类比是令人宽心的，因为这两段时期意味着典型的周期性衰退。
...

# raw.en
“We can’t waste time,” he says.
Yet, according to the political economist Moeletsi Mbeki, at his core, “Zuma is a conservative.”

# norm.en
"We can't waste time," he says.
Yet, according to the political economist Moeletsi Mbeki, at his core, "Zuma is a conservative."

# norm.zh
1929年还是1989年?
巴黎-随着经济危机不断加深和蔓延，整个世界一直在寻找历史上的类似事件希望有助于我们了解目前正在发生的情况。
一开始，很多人把这次危机比作1982年或1973年所发生的情况，这样得类比是令人宽心的，因为这两段时期意味着典型的周期性衰退。

# norm.seg.zh
1929 年 还是 1989 年 ?
巴黎 - 随着 经济危机 不断 加深 和 蔓延 ， 整个 世界 一直 在 寻找 历史 上 的 类似 事件 希望 有助于 我们 了解 目前 正在 发生 的 情况 。
一 开始 ， 很多 人 把 这次 危机 比作 1982 年 或 1973 年 所 发生 的 情况 ， 这样 得 类比 是 令人 宽心 的 ， 因为 这 两段 时期 意味着 典型 的 周期性 衰退 。

# norm.seg.zh
目前 的 趋势 是 ， 要么 是 过度 的 克制 （ 欧洲   ）   ，   要么 是 努力 的 扩展 （ 美国   ）   。
而 历史 是 不 公平 的 。   尽管 美国 要 为 当今 的 全球 危机 负 更 大 的 责任 ， 但 美国 可能 会 比 大多数 国家 以 更 良好 的 势态 走出 困境 。

# norm.seg.tok.zh
目前 的 趋势 是 ， 要么 是 过度 的 克制 （ 欧洲 ） ， 要么 是 努力 的 扩展 （ 美国 ） 。
而 历史 是 不 公平 的 。 尽管 美国 要 为 当今 的 全球 危机 负 更 大 的 责任 ， 但 美国 可能 会 比 大多数 国家 以 更 良好 的 势态 走出 困境 。

# norm.en
"We can't waste time," he says.
Of course, the fall of the house of Lehman Brothers has nothing to do with the fall of the Berlin Wall.
Second, Zoellick should ask why the Bank spends only 2.5% of its budget on the "knowledge bank" research function that it trumpets so proudly in its external relations materials, while it spends three times that amount on maintaining its executive board.

# norm.tok.en
" We can 't waste time , " he says .
Of course , the fall of the house of Lehman Brothers has nothing to do with the fall of the Berlin Wall .
Second , Zoellick should ask why the Bank spends only 2.5 % of its budget on the " knowledge bank " research function that it trumpets so proudly in its external relations materials , while it spends three times that amount on maintaining its executive board .

# norm.tok.en
PARIS - As the economic crisis deepens and widens , the world has been searching for historical analogies to help us understand what has been happening .
At the start of the crisis , many people likened it to 1982 or 1973 , which was reassuring , because both dates refer to classical cyclical downturns .
When the TTIP was first proposed , Europe seemed to recognize its value .
Europe is being cautious in the name of avoiding debt and defending the euro , whereas the US has moved on many fronts in order not to waste an ideal opportunity to implement badly needed structural reforms .

# norm.tok.true.en
Paris - As the economic crisis deepens and widens , the world has been searching for historical analogies to help us understand what has been happening .
at the start of the crisis , many people likened it to 1982 or 1973 , which was reassuring , because both dates refer to classical cyclical downturns .
when the TTIP was first proposed , Europe seemed to recognize its value .
Europe is being cautious in the name of avoiding debt and defending the euro , whereas the US has moved on many fronts in order not to waste an ideal opportunity to implement badly needed structural reforms .

# norm.seg.tok.zh
从 一流 的 麻省理工学院 的 媒体 实验室 到 哈佛大学 的 数学 和 经济系 ， 亚洲 人 - 尤其 是 中国 和 印度人 - 到处 都 是 ， 犹如 公元前 一 世纪 在 雅典 的 罗马 人 一样 ： 他们 对 那里 学到 太 多 东西 的 人们 充满 了 敬佩 ， 而 他们 将 在 今后 几十年 打败 他们 学习 的 对象 。
这 不仅 加大 了 预防 危机 的 难度 - - 尤其 因为 它 为 参与者 提供 了 钻空子 和 逃避责任 的 机会 - - 还 使得 人们 越来越 难以 采取措施 来 应对 危机 。
它们 将 通胀 目标 设定 在 2 % 左右 - - 这 意味着 当 波涛汹涌 时 他们 根本 没有 多少 施展 空间 。

# norm.seg.tok.bpe.zh
从 一流 的 麻省理工学院 的 媒体 实验室 到 哈佛大学 的 数学 和 经济@@ 系 ， 亚洲 人 - 尤其 是 中国 和 印度人 - 到处 都 是 ， 犹如 公元前 一 世纪 在 雅典 的 罗马 人 一样 ： 他们 对 那里 学到 太 多 东西 的 人们 充满 了 敬佩 ， 而 他们 将 在 今后 几十年 打败 他们 学习 的 对象 。
这 不仅 加大 了 预防 危机 的 难度 - - 尤其 因为 它 为 参与者 提供 了 钻@@ 空子 和 逃避@@ 责任 的 机会 - - 还 使得 人们 越来越 难以 采取措施 来 应对 危机 。
它们 将 通胀 目标 设定 在 2 % 左右 - - 这 意味着 当 波@@ 涛@@ 汹涌 时 他们 根本 没有 多少 施展 空间 。

# norm.tok.true.en
indeed , on the surface it seems to be its perfect antithesis : the collapse of a wall symbolizing oppression and artificial divisions versus the collapse of a seemingly indestructible and reassuring institution of financial capitalism .
as a visiting professor at Harvard and MIT , I am getting a good preview of what the world could look like when the crisis finally passes .
one senses something like the making of an American-Asian dominated universe .

# norm.tok.true.bpe.en
indeed , on the surface it seems to be its perfect anti@@ thesis : the collapse of a wall symboli@@ zing oppression and artificial divisions versus the collapse of a seemingly inde@@ struc@@ tible and reassuring institution of financial capitalism .
as a visiting professor at Harvard and MIT , I am getting a good pre@@ view of what the world could look like when the crisis finally passes .
one senses something like the making of an American-@@ Asian dominated universe .

├── data
    └── v15news
        ...
        ├── test.en
        ├── test.zh
        ├── train.en
        ├── train.zh
        ├── valid.en
        └── valid.zh

fairseq-preprocess --source-lang ${src} --target-lang ${tgt} \
    --trainpref ${data_dir}/train --validpref ${data_dir}/valid --testpref ${data_dir}/test \
    --destdir ${data_dir}/data-bin

├── data
    └── v15news
        ...
        └── data-bin
            ├── dict.zh
            ├── dict.en
            ├── preprocess.log
            ├── train.zh-en.zh.idx
            ...
            └── valid.zh-en.en.bin

CUDA_VISIBLE_DEVICES=0,1,2,3 nohup fairseq-train ${data_dir}/data-bin --arch transformer \
	--source-lang ${src} --target-lang ${tgt}  \
    --optimizer adam  --lr 0.001 --adam-betas '(0.9, 0.98)' \
    --lr-scheduler inverse_sqrt --max-tokens 4096  --dropout 0.3 \
    --criterion label_smoothed_cross_entropy  --label-smoothing 0.1 \
    --max-update 200000  --warmup-updates 4000 --warmup-init-lr '1e-07' \
    --keep-last-epochs 10 --num-workers 8 \
	--save-dir ${model_dir}/checkpoints &

epoch 020 | valid on 'valid' subset | loss 4.366 | nll_loss 2.652 | ppl 6.29 | wps 50387.3 | wpb 8026 | bsz 299.8 | num_updates 14400 | best_loss 4.366
epoch 021 | valid on 'valid' subset | loss 4.36 | nll_loss 2.647 | ppl 6.27 | wps 51992.7 | wpb 8026 | bsz 299.8 | num_updates 15120 | best_loss 4.36
epoch 022 | valid on 'valid' subset | loss 4.361 | nll_loss 2.644 | ppl 6.25 | wps 49009.9 | wpb 8026 | bsz 299.8 | num_updates 15840 | best_loss 4.36
epoch 023 | valid on 'valid' subset | loss 4.369 | nll_loss 2.65 | ppl 6.28 | wps 51878.9 | wpb 8026 | bsz 299.8 | num_updates 16560 | best_loss 4.36
epoch 023 | valid on 'valid' subset | loss 4.369 | nll_loss 2.65 | ppl 6.28 | wps 51878.9 | wpb 8026 | bsz 299.8 | num_updates 16560 | best_loss 4.36

├── models
    └── v15news
        ...
        └── checkpoints
            ├── checkpoint40.pt
            ...
            ├── checkpoint49.pt
            ├── checkpoint_best.pt
            └── checkpoint_last.pt

fairseq-generate ${data_dir}/data-bin \
    --path ${model_dir}/checkpoints/checkpoint_best.pt \
    --batch-size 128 --beam 8 > ${data_dir}/result/bestbeam8.txt

S-537	西班牙 的 人权 困境
T-537	Spain 's Human-Rights Dilemma
H-537	-0.16863664984703064	Spain 's Human Rights Quandary
D-537	-0.16863664984703064	Spain 's Human Rights Quandary
P-537	-0.0973 -0.1385 -0.1464 -0.0123 -0.4252 -0.4299 -0.0110 -0.0884

S-5516	这是 不可 接受 的 。
T-5516	that is unacceptable .
H-5516	-0.35840675234794617	this is unacceptable .
D-5516	-0.35840675234794617	this is unacceptable .
P-5516	-0.7625 -0.5517 -0.2005 -0.1513 -0.1261

S-676	与 最初 版本 的 破产法 相 比较 ， 2006 年 的 法律 是 牢牢 扎根 于 市场经济 的 。
T-676	compared with the original bankruptcy code , the 2006 code is firmly rooted in the needs of a market economy .
H-676	-0.624997079372406	in contrast to the original bankruptcy law , the law of 2006 was firmly rooted in the market economy .
D-676	-0.624997079372406	in contrast to the original bankruptcy law , the law of 2006 was firmly rooted in the market economy .
P-676	-1.4995 -0.9434 -0.1292 -0.3479 -0.9758 -0.6600 -0.9037 -0.1836 -0.4983 -1.6406 -0.3142 -0.0344 -0.1685 -1.0289 -1.0286 -0.1917 -1.5369 -0.6586 -0.1119 -0.1333 -0.1361

S-432	用 缅因州 共和党 参议员 苏珊 · 柯林斯 （ Susan Collins ） 的话 说 ， 政府 关门 对 其 缅因州 阿卡迪亚 国家 公园 （ Acadia National Park ） 周边 " 所有 小企业 都 造成 了 伤害 " ， " 这是 完全 错误 的 。 " 是 她 首先 提出 了 和解 协议 纲要 并 送交 参议院 。
T-432	in the words of Senator Susan Collins , a Republican from Maine who first put together the outlines of a deal and took it to the Senate floor , the shutdown " hurt all the small businesses " around Acadia National Park in her home state , " and that is plain wrong . "
H-432	-0.7003933787345886	in the words of Susan Collins , a Republican senator from Maine , it would be a mistake to shut down the government 's " all small business " around the Maine National Park , where she proposed a settlement and delivered it to the Senate .
D-432	-0.7003933787345886	in the words of Susan Collins , a Republican senator from Maine , it would be a mistake to shut down the government 's " all small business " around the Maine National Park , where she proposed a settlement and delivered it to the Senate .
P-432	-1.2762 -0.3546 -0.0142 -0.1261 -0.0058 -0.7617 -0.1695 -0.2992 -0.0777 -0.3016 -0.4818 -0.0061 -0.0308 -0.3509 -2.5533 -1.5254 -0.2761 -1.1667 -0.6169 -0.6285 -1.2463 -0.0973 -1.4414 -0.3324 -0.2302 -0.3312 -0.6847 -1.0005 -0.1812 -2.9048 -0.3072 -1.8045 -0.0473 -0.8421 -0.4715 -0.6841 -1.1902 -1.6192 -0.3370 -2.3317 -0.3701 -0.2508 -3.0284 -0.2336 -1.1318 -0.3904 -0.1124 -0.0262 -0.2203 -0.1480

fairseq-interactive ${data_dir}/data-bin \
    --input ${data_dir}/test.zh \
    --path ${model_dir}/checkpoints/checkpoint_best.pt \
    --batch-size 1 --beam 8 --remove-bpe > ${data_dir}/result/bestbeam8.txt

# predict.tok.true.bpe.en
the subsidy .
this is unacceptable .
there is even worse .
this must change .

# answer.tok.true.bpe.en
removal of subsidies .
that is unacceptable .
it gets worse .
this must change .

# answer.tok.true.bpe.en
a World of Under@@ investment
that needs to change .
revolts of the Righ@@ teous
Russia 's Economic Imperi@@ alism
shock and Pan@@ ic

# answer.tok.true.en
a World of Underinvestment
that needs to change .
revolts of the Righteous
Russia 's Economic Imperialism
shock and Panic

# predict.tok.true.en
the subsidy .
this is unacceptable .
there is even worse .
this must change .

# predict.tok.en
The subsidy .
This is unacceptable .
There is even worse .
This must change .

BLEU = 28.81, 61.8/35.4/22.8/15.2 (BP=0.976, ratio=0.977, hyp_len=187605, ref_len=192093)
It is not advisable to publish scores from multi-bleu.perl.  The scores depend on your tokenizer, which is unlikely to be reproducible from your paper or consistent across research groups.  Instead you should detokenize then use mteval-v14.pl, which has a standard tokenization.  Scores from multi-bleu.perl can still be used for internal purposes when you have a consistent tokenizer.

# predict.tok.en
what 's wrong with protectionism ?
The " establishment " and counterinsurgency strategy , introduced by President Barack Obama 's military surge in 2010 , was intended to reverse the war .

# predict.en
What's wrong with protectionism?
The "establishment" and counterinsurgency strategy, introduced by President Barack Obama's military surge in 2010, was intended to reverse the war.

CUDA_VISIBLE_DEVICES=0,1,2,3 fairseq-train ...

Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
	Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
Traceback (most recent call last):
  ...
  site-packages/torch/multiprocessing/spawn.py", line 111, in join
    raise Exception(
Exception: process 2 terminated with exit code 1