Speech Pattern Recognition using Neural Networks
4.2 Bayes Decision Theory .1 Preparations
4.2.5 Discriminative Training
# OQTG FKTGEV CRRTQCEJ VJCP VJG /.' CPF $C[GUKCP OGVJQFU VQ C UWEEGUUHWN GZG EWVKQP QH VJG ENCUUKſECVKQP FGEKUKQP TGVWTPU VQ 6JG CRRTQCEJ QTKIKPCVGF KP VJG ENCUUKE VTCKPKPI OGVJQF ECNNGF NKPGCT FKUETKOKPCPV CPCN[UKU CPF KV KU WUWCNN[ TGHGTTGF VQ CU FKUETKOKPCVKXG VTCKPKPI +V GOWNCVGU VJG GPVKTG ENCUUKſECVKQP FKUETKOKPCVKQP QR GTCVKQP QH CPF CVVGORVU VQ TGCNK\G C UGV QH FKUETKOKPCPV HWPEVKQPU EQPUGSWGPVN[
VJCV CEJKGXG VJG FGUKTCDNG KP VGTOU QH C RTGUGV VTCKPKPI QDLGEVKXG ETKVGTKQP ENCU UKſECVKQP RGTHQTOCPEG HQT VTCKPKPI UCORNGU
6JG MG[ FGUKIP KUUWGU VJCV FGVGTOKPG VJG RGTHQTOCPEG QH FKUETKOKPCVKXG VTCKPKPI KP ENWFG VJG HQNNQYKPI JQY VQ FGſPG VJG FKUETKOKPCPV HWPEVKQPU JQY VQ GXCNWCVG VJG RGTHQTOCPEG QH FGEKUKQPU KP VJG VTCKPKPI UVCIG JQY VQ CFLWUV VTCKPCDNG U[UVGO RCTCOGVGTU GI CPF JQY VQ EQRG YKVJ WPMPQYP UCORNGU VJCV FQ PQV CRRGCT KP VJG VTCKPKPI UVCIG +V VWTPU QWV VJCV VJGUG RQKPVU EQORTGJGPUKXGN[ EQXGT VJG ENCUUKſGT FGUKIP RTQEGFWTG 6JGUG KUUWGU YKNN DG FGUETKDGF KP VWTP KP NCVGT RCIGU
4.2.5.1 Functional Form Embodiment of the Entire Process
'OWNCVKQP QH VJG ENCUUKſECVKQP RTQEGUU KPGXKVCDN[ TGSWKTGU VJCV VJG RTQEGUU DG TGRTG UGPVGF KP C VTCEVCDNG HWPEVKQPCN HQTO YJKEJ GPCDNGU QPG VQ FGCN YKVJ VJG FGUKIP QH KVU EQTTGURQPFKPI ENCUUKſECVKQP U[UVGO OCVJGOCVKECNN[ $CUGF QP CP QDUGTXCVKQP QH VJG FGEKUKQP RTQEGUU KP QPG ECP UGG VJCV VJG FGEKUKQP EQPUKUVU QH VJG EQORCTKUQP QH VJG FKUETKOKPCPV HWPEVKQPU QXGT CNN RQUUKDNG ENCUUGU # IGPGTCN YC[ QH GOWNCVKPI VJG RTQEGUU KU IKXGP KP VJG HQNNQYKPI HWPEVKQPCN HQTO YJKEJ KU QHVGP TGHGTTGF VQ CU C OKUENCUUKſECVKQP OGCUWTG
YJGTGKU C RQUKVKXG EQPUVCPV 1PG ECP PQVKEG JGTG VJCV IKXGP QH
KPFKECVGU C OKUENCUUKſECVKQP CPF
KPFKECVGU C EQTTGEV ENCUUKſECVKQP +P CFFKVKQP EQPVTQNNKPIGPCDNGU VJG UKOWNCVKQP QH XCTKQWU FGEKUKQP TWNGU +P RCTVKEWNCT YJGPCRRTQCEJGU ENQUGN[ GOWNCVGU TWNG
4.2.5.2 Discriminant Functions
6JG UGNGEVKQP QH C HWPEVKQPCN HQTO QH VJG FKUETKOKPCPV HWPEVKQP KU DCUKECNN[ GSWKXCNGPV VQ VJG UGNGEVKQP QH C OGCUWTG QT OGCUWTGOGPV WUGF VQ TGRTGUGPV VJG FGITGG VQ YJKEJ
CP KPRWV UCORNG DGNQPIU VQ UQOG ENCUU +P CFFKVKQP VJG HQTO TGNKGU QP VJG V[RG QH VTCKPCDNG U[UVGO RCTCOGVGTU KG CPF KV KU IGPGTCNN[ UGNGEVGF DCUGF QP VJG PCVWTG QH VJG RCVVGTPU
6[RKECN GZCORNGU QH ENCUUKECN HQTOU HQT ſZGFFKOGPUKQPCN RCVVGTPU KPENWFG VJG NKP GCT FKUETKOKPCPV HWPEVKQP CPF VJG FKUVCPEG 6JG OGCUWTG WUGF KP VJG NKPGCT FKUETKO KPCPV HWPEVKQP KU C NKPGCTN[YGKIJVGF UWO QH KPRWV XGEVQT EQORQPGPVU YJGTG C UGV QH YGKIJVU EQTTGURQPFU VQ VTCKPCDNG U[UVGO RCTCOGVGTU +P VJG FKUVCPEG ECUG VJG OGCUWTG KU C TGCUQPCDN[ UGNGEVGF FKUVCPEG DGVYGGP CP KPRWV CPF C TGHGTGPEG XGEVQT UWEJ CU 'WENKFGCP FKUVCPEG YJGTG VJG TGHGTGPEG XGEVQTU YQTM CU
(QT VJG ECUG QH ENCUUKH[KPI F[PCOKE RCVVGTPU UWEJ CU URGGEJ UCORNGU U[UVGO RCTCO GVGTU UJQWNF EQPVCKP UQOG OGCPU QH TGRTGUGPVKPI VJG VGORQTCN UVTWEVWTG # VTCFKVKQPCN UGNGEVKQP GORNQ[U TGHGTGPEG XGEVQTU CPF WUGU C &2DCUGF PQTOCNK\GF FKUVCPEG OGC UWTG DGVYGGP CP KPRWV RCVVGTP CPF C TGHGTGPEG XGEVQT UGSWGPEG 1P VJG QVJGT JCPF KP OQUV TGEGPV ECUGU VTCKPCDNG U[UVGO RCTCOGVGTU CTG VJG EQORQPGPV RTQDCDKNKVKGU QH *// UWEJ CU UVCVG VTCPUKVKQP RTQDCDKNKV[ CPF QDUGTXCVKQP GOKUUKQP RTQDCDKNKV[
6JG VGORQTCN UVTWEVWTG KU TGRTGUGPVGF VJGTGKP CU VJG UVCVG VTCPUKVKQP UVTWEVWTG +P VJG ECUGU WUKPI 00U C FKUETKOKPCPV HWPEVKQP KU DCUKECNN[ CP QWVRWV QH 00ŏU QWVRWV PQFG 6JG V[RG QH OGCUWTG VJGTGKP TGNKGU QP VJG UGNGEVKQP QH PQFG HWPEVKQPU UWEJ CU RGTEGRVTQPNKMG CEVKXCVKQP HWPEVKQP TCFKCNDCUKU CEVKXCVKQP HWPEVKQP CPF UKIOQKFCN QWVRWV HWPEVKQP
4.2.5.3 Loss over an Individual Pattern
6Q GXCNWCVG FGEKUKQP RGTHQTOCPEG KP VJG FGUKIP UVCIG KPFKXKFWCN NQUU KU KPVTQFWEGF HQT GXGT[ VTCKPKPI RCVVGTP YJKEJ KU C HWPEVKQP QH VJG OKUENCUUKſECVKQP OGCUWTG CPF TGƀGEVU ENCUUKſECVKQP FGITCFCVKQP +V UJQWNF DG PQVGF VJCV KPFKXKFWCN NQUU KU UQOG VKOGU TGHGTTGF VQ CU KPFKXKFWCN TKUM QT KPFKXKFWCN FGUKIP QDLGEVKXG 1DXKQWUN[ VJG UOCNNGT VJG NQUU KU VJG OQTG FGUKTCDNG KVU EQTTGURQPFKPI FGEKUKQP KU # PCVWTCN CPF IGPGTCN HWPEVKQPCN HQTO QH NQUU KU IKXGP HQT URGGEJ RCVVGTPQHCU
YJGTGKU C UECNCT HWPEVKQP VJCV FGVGTOKPGU VJG EJCTCEVGTKUVKEU QH VJG NQUU HWPEVKQP CPFCPF CTG EQPUVCPVU (WTVJGTOQTG KU QH VJG UKIOQKFCN HQTO YJKEJ KU MPQYP CU C UOQQVJGF NQIKUVKE HWPEVKQP CPF KV KU CNUQ C UOQQVJGF XGTUKQP QH VJG ENCUUKſECVKQP GTTQT EQWPV
QVJGTU
4.2.5.4 Loss over Multiple Patterns
6Q FGVGTOKPG VJG RGTHQTOCPEG QH C FGUKIPGF ENCUUKſGT QXGT OWNVKRNG RCVVGTP UCORNGU VJG KPFKXKFWCN NQUU KU PCVWTCNN[ CRRNKGF KP VJG HQNNQYKPI GORKTKECN CXGTCIG NQUU HQTO
VQ C ſPKVG DWV WUWCNN[ NCTIG UGV QH VTCKPKPI RCVVGTP UCORNGU YJGTG KU VJG PWODGT QH VTCKPKPI UCORNGU KP CPF QH GZRNKEKVN[ OGCPU VJCV VJG UCORNG KU VJGVJ UCORNG QH #UUWOKPI VJG KPFKXKFWCN NQUU VQ DG VJKU GORKTKECN CXGTCIG NQUU DGEQOGU C UOQQVJGF XGTUKQP QH VJG VQVCN EQWPV QH ENCUUKſECVKQP GTTQTU OGCUWTGF QXGT
4.2.5.5 Adjustment of Trainable System Parameters
7UKPI VJG GORKTKECN CXGTCIG NQUU VJG FGUKIP RTQEGGFU VQ CP CEVWCN VTCKPKPI RTQEGFWTG QT KP QVJGT YQTFU VJG CFLWUVOGPV QH VTCKPCDNG U[UVGO RCTCOGVGTU # IQCN QH VJG CFLWUVOGPV KU QDXKQWUN[ VQ CEJKGXG VJG UVCVWU QH ENCUUKſGT RCTCOGVGTKP QWT ECUG VJCV TGUWNVU KP VJG OKPKOK\CVKQP QH VJG GORKTKECN CXGTCIG NQUU
7UWCNN[ VJG HWPEVKQPCN HQTO QH VJG GORKTKECN CXGTCIG NQUU KU WPMPQYP CPF KV KU VJWU TCTGN[ RQUUKDNG VQ CEJKGXG VJG VTWG INQDCN OKPKOWO UVCVWU QH VJG NQUU CPCN[VKECNN[
#EEQTFKPIN[ KP OQUV ECUGU VJG CFLWUVOGPV KU HQTOWNCVGF CU CP CU[ORVQVKECN VTCKP KPI OGVJQF VJCV DCUKECNN[ IWCTCPVGGU ſPFKPI CV OQUV NQECN OKPKOC QH VJG NQUU QT VJG INQDCN OKPKOWO QH VJG NQUU QPN[ KP VJG RTQDCDKNKUVKE UGPUG 6[RKECN GZCORNGU QH HQTOWNCVGF OGVJQFU KPENWFG VJQUG DCUGF QP VJG UVGGRGUV FGUEGPV OGVJQF VJG RTQDC DKNKUVKE FGUEGPV OGVJQF UKOWNCVGF CPPGCNKPI CPF IGPGVKE CNIQTKVJOU #OQPI VJGUG YG KPVTQFWEG CP CFLWUVOGPV OGVJQF DCUGF QP VJG RTQDCDKNKUVKE FGUEGPV OGVJQF =?
YJKEJ IKXGU C IGPGTCN OCVJGOCVKECN HTCOGYQTM QH CFCRVKXG UGSWGPVKCN QT UCORNG D[UCORNG RCTCOGVGT CFLWUVOGPV
# MG[ EQPEGRV QH VJG RTQDCDKNKUVKE FGUEGPV OGVJQF KU VJCV IKXGP VJG NQUU UWTHCEG HWPE VKQP VJG TGRGVKVKQP QH UOCNNUVGR FGUEGPV QRGTCVKQPU NGCFU CV NGCUV VQ C NQECN OKPKOWO RQKPV QH VJG UWTHCEG KP VJG RTQDCDKNKUVKE UGPUG JGTG VJG GPVKTG UJCRG QH VJG UWTHCEG KU WPQDUGTXCDNG #P CFLWUVOGPV TWNG DCUGF QP VJKU EQPEGRV KU UWOOCTK\GF KP VJG HQNNQYKPI VJGQTGO
[Probabilistic Descent Theorem]
#UUWOG VJCV C VTCKPKPI UCORNG KU IKXGP CV VTCKPKPI VKOG KPFGZ +H VJG ENCUUKſGT RCTCOGVGT CFLWUVOGPVÆKU URGEKſGF D[
Æ
Í
CPF C UGSWGPEG QH RQUKVKXG TGCN PWODGTU KP TGHGTTGF VQ CU NGCTPKPI YGKIJVU UCVKUſGU
VJGP VJG RCTCOGVGT CFLWUVOGPV CEEQTFKPI VQ
Æ
EQPXGTIGU YKVJ RTQDCDKNKV[ QPG CV NGCUV VQ YJKEJ TGUWNVU KP C NQECN OKPKOWO QH YJKEJ KU VJG GZRGEVGF NQUU FGſPGF CU HQNNQYU
YJGTGTGRTGUGPVU VJG UVCVG QHCVÍKU C RQUKVKXGFGſPKVG OCVTKZ CPFKU VJG GPVKTG UCORNG URCEG QH VJGRCVVGTPU +V KU CUUWOGF VJCV
CPF VJCVKU CP KPFKECVQT HWPEVKQP
6JG KORQTVCPV RQKPVU JGTG CTG VJCV VJG CFLWUVOGPV CNYC[U CVVGORVU VQ TGOQXG VJG ENCUUKſECVKQP GTTQT ECWUGF D[ C PGYN[ RTGUGPVGF VTCKPKPI UCORNG CPF CUUWOKPI VJG KPFKXKFWCN NQUU VQ DG VJG CFLWUVOGPV ECP TGUWNV KP VJG NQECN OKPKOK\CVKQP QH VJG GZRGEVGF ENCUUKſECVKQP GTTQT EQWPV NQUU KP VJG RTQDCDKNKUVKE UGPUG 6JG VJGQTGO FQGU PQV IWCTCPVGG VJG WPEQPFKVKQPCN OKPKOK\CVKQP QH VJG GZRGEVGF NQUU KG VJG CEJKGXGOGPV QH VJG OKPKOWO ENCUUKſECVKQP GTTQT RTQDCDKNKV[ EQPFKVKQP *QYGXGT KV ENGCTN[ VWTPU QWV VJCV VJG FKUETKOKPCVKXG VTCKPKPI FGUETKDGF KP VJG CDQXG RCTCITCRJU OQTG ENGCTN[ TGUGODNGU VJG $C[GU FGEKUKQP TWNG VJCP VJG RTQDCDKNKV[ HWPEVKQP GUVKOCVKQP CRRTQCEJGU UWEJ CU VJG OCZKOWO NKMGNKJQQF OGVJQF
4.2.5.6 Training Optimality
6JG WNVKOCVG IQCN QH ENCUUKſGT FGUKIP KU VQ ſPF VJG ENCUUKſGT RCTCOGVGT UGV VJCV CEJKGXGU VJG OKPKOWO ENCUUKſECVKQP GTTQT RTQDCDKNKV[ EQPFKVKQP *QYGXGT CU EKVGF CDQXG RCVVGTPU CXCKNCDNG HQT VTCKPKPI CTG WUWCNN[ ſPKVG CPF VJWU KV KU FKHſEWNV VQ FK TGEVN[ CKO CV VJG WNVKOCVG IQCN YJKEJ YQWNF KPGXKVCDN[ TGSWKTG VJG EQORNGVG EQO RWVCVKQP QH VJG RTQDCDKNKV[ TGNCVGF VQ VJG UCORNG FKUVTKDWVKQP QT KVU EQTTGURQPFKPI QDUGTXCVKQP QH CNN RQUUKDNG RCVVGTP UCORNGU #EVWCNN[ VTCKPKPI WUKPI C NKOKVGF PWO DGT QH UCORNGU DCUKECNN[ NGCFU VQ CV OQUV C OKPKOWO ENCUUKſECVKQP GTTQT EQPFKVKQP QXGT VJG ſPKVG RCVVGTP UCORNGU CPF VJKU FGUETKDGU NKVVNG KP C OCVJGOCVKECNN[ TKIQTQWU UGPUG CDQWV VJG ENCUUKſECVKQP RGTHQTOCPEG QXGT WPMPQYP HWVWTG RCVVGTPU 6JGTGHQTG KP QTFGT VQ DTKFIG VJG WNVKOCVG IQCN KG VJG OKPKOWO ENCUUKſECVKQP GTTQT RTQDCDKN KV[ EQPFKVKQP CPF VJG RTCEVKECN FGUKIP CVVGORVU CPCN[UGU QH VJG VTCKPKPI QRVKOCNKV[
QXGT KPſPKVG VTCKPKPI UCORNGU CTG PGGFGF VQ UQOG GZVGPV GXGP VJQWIJ VJG[ CTG QPN[
VJGQTGVKECN
(QT GZRNCPCVKQP RWTRQUGU NGV WU CUUWOG VJCV C RTQDCDKNKV[ OGCUWTGKU RTQ XKFGF KP C MPQYP HWPEVKQPCN HQTO HQT RCVVGTP UCORNG CPF C RCTCOGVGT UGV FG VGTOKPKPI VJG HWPEVKQPCN HQTO KU 6JGP EQPUKFGTKPI VJG FKUETKOKPCPV HWPEVKQP
CPF VJG OKUENCUUKſECVKQP OGCUWTG QH YG ECP TGYTKVG VJG GZRGEVGF NQUU VJCV KU
FGſPGF D[ WUKPI VJG UOQQVJ ENCUUKſECVKQP GTTQT EQWPV NQUU CU 1DXKQWUN[ VJG NCUV GZRTGUUKQP KU GSWKXCNGPV VQ VJG GZRGEVGF GTTQT ECWUGF D[ VJG ENCU UKſECVKQP FGEKUKQP WUKPI VJG GUVKOCVGU QH VJG C RQUVGTKQTK RTQDCDKNKVKGU 0QVG JGTG VJCV VJG RCTCOGVTKE HQTO KU MPQYP DWV KV KU PQV MPQYP YJGVJGT VJG RTGUGPV UVC VWU QH TGUWNVU KP VJG OKPKOWO GTTQTTCVG ENCUUKſECVKQP *GTG VJG FKHHGTGPEG KP VJG PGCT GSWCNKV[ QH QTKIKPCVGU KP VJG UOQQVJPGUU EQPVCKPGF KP VJG GTTQT EQWPV NQUU
%QPUGSWGPVN[ D[ EQPVTQNNKPI VJG UOQQVJPGUU QH HWPEVKQPU UWEJ CU VJGPQTO WUGF KP CPF VJG UKIOQKFCN HWPEVKQP WUGF KP YG ECP CTDKVTCTKN[ OCMG ENQUGT VQ VJG NCUV GSWCVKQP KP +V UJQWNF DG PQVGF JGTG VJCV VJGTG KU C NKPM DGVYGGP VJG RTCEVKECN FGUKIP QRGTCVKQP DCUGF QP VJG FKUETKOKPCVKXG VTCKPKPI CPF VJG EQORWVCVKQP QH VJG GZRGEVGF NQUU YJKEJ KU VJG WNVKOCVG VJGQTGVKECN IQCN QH ENCUUKſGT FGUKIP 0GZV NGV WU TGECNN VJCV YG WUG YJQUG HWPEVKQPCN HQTO KU CUUWOGF VQ DG MPQYP
$CUGF QP VJKU HCEV CPF CNUQ VJG TGNCVKQP DGVYGGP VJG OKPKOWO GTTQTTCVG ENCUUKſECVKQP CPF KVU EQTTGURQPFKPI C RQUVGTKQTK RTQDCDKNKVKGU VJG UVCVWU QH VJCV EQTTGURQPFU VQ VJG OKPKOWO QH KP YJKEJ KU CEJKGXGF D[ CFLWUVKPI KU ENGCTN[ GSWCN VQ VJGVJCV CEJKGXGU VJG OCZKOWO C RQUVGTKQTK RTQDCDKNKV[ EQPFKVKQP +P UJQTV KV VWTPU QWV VJCV VJG OKPKOWO EQPFKVKQP QH ECP DGEQOG CTDKVTCTKN[ ENQUG VQ VJG KFGCN OKPKOWO GTTQTTCVG EQPFKVKQP VJCV KU CUUQEKCVGF D[ VJG OKPKOWO ENCUUKſECVKQP GTTQT RTQDCDKNKV[
YJGTG KU C RCTVKCN URCEG QHVJCV ECWUGU C ENCUUKſECVKQP GTTQT CEEQTFKPI VQ VJG OCZKOWO C RQUVGTKQTK RTQDCDKNKV[ FGEKUKQP TWNG KG
6JG CDQXG CPCN[UKU QH VJG OKPKOWO GTTQT EQPFKVKQP KU KORQTVCPV HTQO VJG VJGQTGVK ECN XKGYRQKPV QH UJQYKPI VJG TCVKQPCNKV[ QH VJG FKUETKOKPCVKXG VTCKPKPI HQTOCNKUO +V RTQXKFGU C OCVJGOCVKECNN[ UQWPF DCEMITQWPF VQ RTCEVKECN CVVGORVU CV ENCUUKſGT FG UKIP DCUGF QP FKUETKOKPCVKXG VTCKPKPI +P TGCNKV[ JQYGXGT VJG RCTCOGVGT UGV KU TCTGN[ MPQYP CPF VJWU KV KU WUWCNN[ KORQUUKDNG VQ CEJKGXG VJG OKPKOWO ENCUUKſEC VKQP GTTQTTCVG EQPFKVKQP VJTQWIJ FKUETKOKPCVKXG VTCKPKPI QT KP QVJGT YQTFU VJG NQUU OKPKOK\CVKQP QXGT ſPKVG VTCKPKPI UCORNGU
4.2.5.7 Global Design Scope
9G JCXG CUUWOGF VJCV C HGCVWTG GZVTCEVKQP OQFWNG KU FGVGTOKPGF UGRCTCVGN[ HTQO VJG FGUKIP QH C ENCUUKſGT DWV VJKU CUUWORVKQP KU QPN[ CFQRVGF VQ UKORNKH[ FKUEWUUKQP
#EVWCNN[ HGCVWTG GZVTCEVQTU CTG WUWCNN[ FGUKIPGF DCUGF QP UEKGPVKſE GZRGTVKUG CPF UWEJ UGRCTCVG CPF GORKTKECN FGUKIP KU C UVCPFCTF CRRTQCEJ VQ HGCVWTG GZVTCEVQT FGUKIP CU C TGUWNV TGEQIPK\GT FGUKIP *QYGXGT HTQO VJG HQTOCNKUO KPVTQFWEGF CDQXG QPG ECP GCUKN[ EQPENWFG VJCV VJG FGUKIP UEQRG QH VJG FKUETKOKPCVKXG VTCKPKPI HQT C DCEMGPF ENCUUKſGT ECP DG GZVGPFGF VQ KVU EQTTGURQPFKPI HTQPVGPF HGCVWTG GZVTCEVQT # MG[ KP VJKU GZVGPUKQP KU ENGCTN[ VQ WUG VJG EJCKP TWNG QH ECNEWNWU 4GECNN VJCV VJKU TWNG RNC[U C EGPVTCN TQNG KP VJG FGſPKVKQPU QH VJG OKUENCUUKſECVKQP OGCUWTG CPF VJG KPFKXKFWCN NQUU
(TQO VJG FGſPKVKQP QH QWT TGEQIPK\GT UVTWEVWTG C FKUETKOKPCPV HWPEVKQP HQT VJG GPVKTG TGEQIPK\GT ECP DG FGſPGF CU
YJGTGKU CP QWVRWV QH VJG HGCVWTG GZVTCEVKQP OQFWNG %NGCTN[ VJG CFLWUVOGPV VJCV KU RGTHQTOGF HQT VJG NQUU OKPKOK\CVKQP CV VJG NGXGN QH
KU GCUKN[ RTQRCICVGF VQ VJG NGXGN QH NGCFKPI VQ C INQDCN QRVKOK\CVKQP QH VJG GPVKTG U[UVGO