Sunday, January 1, 2017

Merging multiple sensory modalities in deep machine learning

I am studying David Harwath and Jim Glass's paper from CSAIL, "Deep Multimodal Semantic Embeddings for Speech and Images."  I discussed similar work in my 2011 paper "Experiments with Asa H" but rather than speech and images being the sole modalities I allowed for input from a number of Asa's various senses. (The paper is available on my website under "book", chapter 1, page 13.)

Such merging of multiple modalities may explain why it has not been possible to discover a formal grammar that covers human language.

No comments:

Post a Comment