Merging multiple sensory modalities in deep machine learning

I am studying David Harwath and Jim Glass's paper from CSAIL, "Deep Multimodal Semantic Embeddings for Speech and Images."  I discussed similar work in my 2011 paper "Experiments with Asa H" but rather than speech and images being the sole modalities I allowed for input from a number of Asa's various senses. (The paper is available on my website www.robert-w-jones.com under "book", chapter 1, page 13.)

Such merging of multiple modalities may explain why it has not been possible to discover a formal grammar that covers human language.

Case patches and interpolation

Rule patches (ripple-down-rules) were developed as a way to improve the accuracy/performance of production systems. (See Compton and Jansen, Proc. 2nd Australian Joint A. I. Conference, 1988, pg 292). It's possible to do the same sort of thing with case-based reasoning systems, provided one has logged a sufficient amount of experience. A patching case base takes as input the new input vector along with the nearest matching retrieved case vector and provides as output a prediction of the error in the expected output. One can use the error to attempt to correct/improve the output. This simply amounts to interpolation or case adaption. It can be used in case-based reasoners as well as A.s.a. H.

Senses

I've given Asa H a metal detector wand as an additional sensory input.

Different kinds of truth and the need for scientific pluralism

I have argued that truth is a vector quantity. (See my review of  Theories of Truth: a Critical Introduction, by R. Kirkham, at amazon.com, 25 September 2008.) There are various kinds of truth and these are each components of the truth vector. Things like:
      Valid deduction from true assumptions
       Satisfying a definition
       Agreement with observation
       Usefulness
       Coherence, consistency
       Etc.
There are then different kinds of science making use of the various different ways of assessing truth (Along these lines see B. Latour, An Inquiry into Modes of Existence, Harvard U. Press, 2013). This is a further argument for the need for scientific pluralism.

Beauty

Humans use the value/concept "beauty" in both mate and scientific theory selection.  My artificial intelligence Asa H and most other AIs do not currently have a concept of beauty, see, for example, my blog of 21 September 2010. Dorner's Psi cognitive architecture does incorporate a notion of beauty, modeled as an emotion. (D. Dorner, Bauplan fur eine Seele, Reinbeck: Rowohlt, 1999, pg 373) Perhaps I should try to give Asa H a concept of beauty. I have Bach's microPsi 2 software running in my lab. (See my blog of 2 March 2015.) Several groups have assembled software modules that assess the beauty of humans.  Even if an AI does not need such a value for its decision making it might still prove useful for man-machine communication and understanding. (Another example of reconceptualizing reality.)

Doing science can make you more intelligent

Models of cognition involve a value function, a dopamine circuit in humans. (See, for example, the model in my blog of 1 Sept. 2012) The goal of any intelligence is to maximize rewards. How intelligent you are depends upon how good your value system is.  If you have bad values you make bad decisions and get fewer rewards.  Doing science promotes and develops some of the values which we then apply elsewhere in life, things like evidence-based belief.

Expanding consciousness

Using mobile robots, simulators, and hand coding the network fragment that constitutes Asa H's concept of self (see 21 July 2016 blog for a listing) has now been grown, adding the vectors: smell=(MQ-2 sensor, MQ-3, MQ-4, MQ-5, MQ-6, MQ-7, MQ-8, MQ-9, MQ-135), taste=(pH, salinity), touch=(contact1, contact2,..., force1, force2,..., whisker1, whisker2, ...), see=(camera1, camera2, ..., light sensor, color sensor, IR sensor, ...), temperature=(temperature1, temperature2, ...), pain=(pain1, pain2, ...), sense=(smell, taste, humidity, touch, see, hear, pain, B field, E field/charge, sense far, sense near, motion sensors, sense weight, compass heading, accelerometers, wind, temperature, air pressure, GPS positions, solar power), image manipulation=(shift y, shift x, rotatecw, rotateccw, scale +, scale -, reflect, invert), think=(sort, load, save, simulation, search, deduction, interpolation, extrapolation, update, image manipulation, mutate, sensitivity analysis, forgetting, compression), leave=(sense near, move, sense far), battery charge=(battery1 charge, battery2 charge,...), act=(grasp, release, kick, lift, lower, walk, move, carry, turn, approach, leave, push, pull, twist, recharge, taste, electromagnet, fan, messaging, telemessaging), health=(battery charge, pain, temperature, hardware diagnostic, software diagnostic). The development of hardware and software diagnostics is an ongoing project. All together Asa's concept of its self has roughly doubled in size.