At approximately 17:46 he says "Your're going to have a mixture of both [symbolic & connectionism] .. to have a successful application", what does he mean by that?
Good question (and thanks for the timestamp)! So this could mean a couple of things: in the edge context, I would think it means that since we can't go all the way to deploying extremely large models (requiring lots of on-device compute) on edge devices, you can use smaller/compressed NNs in limited contexts and symbolic methods elsewhere as appropriate. Another way to understand "mixture of symbolic & connectionism" is the class of neuro-symbolic methods (this came up in an interview I did with Chris Manning a while ago!).
Apr 7, 2023·edited Apr 7, 2023Liked by daniel bashir
Thanks for listening and for your question! To add to Daniel's existing comment: I think it's important to always think of "edge AI" in an application context, since it's a set of tools for solving real world problems. While in machine learning theory we often think about models in isolation, when applied to a tangible problem it's very rare that we can use the output of a model by itself.
For example, imagine we're trying to build a system to recognize when the user has said a particular keyword ("hello robot"). An efficient way to do this is to train a deep learning model that will transform processed audio (in the form of spectrograms) into a time series of phonemes. That's the connectionist part. We could then write a simple matching engine to scrutinize the phonemes and their timing, and decide whether we think the keyword has been spoken, so we can do something useful. That's the symbolic part.
While it would certainly be possible to do all of this in a single deep learning model, moving some of the functionality to a symbolic space allows us to do useful things like zero-shot matching of new keywords at runtime (allowing for customization without retraining). It's really convenient to be able to control the system's behavior by tweaking traditional software rather than having to train and deploy a new model.
This is nothing new: embedded systems engineers make great use of things like state machines in order to turn noisy, messy sensor input into something more useful. A good way to think about it is that deep learning models help transform sensor data, but they still emit signals. Traditional programming, including symbolic AI techniques, are helpful in translating those signals into useful state.
At approximately 17:46 he says "Your're going to have a mixture of both [symbolic & connectionism] .. to have a successful application", what does he mean by that?
Good question (and thanks for the timestamp)! So this could mean a couple of things: in the edge context, I would think it means that since we can't go all the way to deploying extremely large models (requiring lots of on-device compute) on edge devices, you can use smaller/compressed NNs in limited contexts and symbolic methods elsewhere as appropriate. Another way to understand "mixture of symbolic & connectionism" is the class of neuro-symbolic methods (this came up in an interview I did with Chris Manning a while ago!).
> neuro-symbolic methods
This is interesting; I will listen to your podcast with Manning!
Thanks for listening and for your question! To add to Daniel's existing comment: I think it's important to always think of "edge AI" in an application context, since it's a set of tools for solving real world problems. While in machine learning theory we often think about models in isolation, when applied to a tangible problem it's very rare that we can use the output of a model by itself.
For example, imagine we're trying to build a system to recognize when the user has said a particular keyword ("hello robot"). An efficient way to do this is to train a deep learning model that will transform processed audio (in the form of spectrograms) into a time series of phonemes. That's the connectionist part. We could then write a simple matching engine to scrutinize the phonemes and their timing, and decide whether we think the keyword has been spoken, so we can do something useful. That's the symbolic part.
While it would certainly be possible to do all of this in a single deep learning model, moving some of the functionality to a symbolic space allows us to do useful things like zero-shot matching of new keywords at runtime (allowing for customization without retraining). It's really convenient to be able to control the system's behavior by tweaking traditional software rather than having to train and deploy a new model.
This is nothing new: embedded systems engineers make great use of things like state machines in order to turn noisy, messy sensor input into something more useful. A good way to think about it is that deep learning models help transform sensor data, but they still emit signals. Traditional programming, including symbolic AI techniques, are helpful in translating those signals into useful state.
Thanks for the detailed explanation!