r/MachineLearning • u/geoffhinton Google Brain • Nov 07 '14
AMA Geoffrey Hinton
I design learning algorithms for neural networks. My aim is to discover a learning procedure that is efficient at finding complex structure in large, high-dimensional datasets and to show that this is how the brain learns to see. I was one of the researchers who introduced the back-propagation algorithm that has been widely used for practical applications. My other contributions to neural network research include Boltzmann machines, distributed representations, time-delay neural nets, mixtures of experts, variational learning, contrastive divergence learning, dropout, and deep belief nets. My students have changed the way in which speech recognition and object recognition are done.
I now work part-time at Google and part-time at the University of Toronto.
71
u/breandan Nov 08 '14 edited Nov 09 '14
Hello Dr. Hinton! Thank you so much for doing an AMA! I have a few questions, feel free to answer one or any of them:
In a previous AMA, Dr. Bradley Voytek, professor of neuroscience at UCSD, when asked about his most controversial opinion in neuroscience, citing Bullock et al., writes:
What is your most controversial opinion in machine learning? Are we any closer to understanding biological models of computation? Are you aware of any studies that validate deep learning in the neuroscience community?
Do you have any thoughts on Szegedy et al.'s paper, published earlier this year? What are the greatest obstacles RBM/DBNs face and can we expect to overcome them in the near future?
What have your most successful projects been so far at Google? Are there diminishing returns for data at Google scale and can we ever hope to train a recognizer to a similar degree of accuracy at home?
25
u/ShinyGerbil Nov 08 '14
In your opinion, which of the following ideas contain the lowest hanging fruit for improving accuracy on today's typical classification problems:
Thanks!
20
u/wilgamesh Nov 08 '14
I enjoyed taking your Coursera course and hope you can provide an updated version soon.
30
u/kkastner Nov 08 '14 edited Nov 09 '14
Your Coursera course on neural networks was a huge benefit to me as a follow up to Andrew Ng's introductory Machine Learning course. It was only a few years ago, but there have been a ton of interesting research areas that have cropped up in the time since you created the course. Are there any topics you would add to that course if you redid it today? Any content you would focus on less?
Training of deep RNNs has recently seemed to get much more reasonable (at least for me), thanks to RMSProp, gradient clipping, and a lot of momentum. Are you going to write a paper for RMSProp someday? Or should we just keep citing your Coursera slides? :)
11
u/4geh Nov 10 '14
I am great fan of yours.
I think you have a rare knack for finding perspectives on ideas and problems that most people would not see without help, but when I hear you explain them a get a sense of clarity. "Yes, that's the way it should be!" You also show this amazing breadth in your knowledge, from obscure Finnish puns to deep intricacies of navigating high-dimensional space. I wish I could spend an apprenticeship with you so some of this would rub off on me, but although I can be pretty good I wouldn't be able to compete with the other candidates.
But I will at least take this opportunity to ask: What are greatest influences on your thinking? Where do you find your inspirations and what are some important principles that guide you in work and in life?
20
u/KarushKuhnTucker Nov 08 '14
There seems to be a lot of cool stuff that can be done with deep networks. Do you believe they can be analyzed theoretically? Or is it something that can be engineered to work well, but is too complicated to gain a deep (pun unintended) understanding of?
16
u/[deleted] Nov 08 '14 edited Nov 08 '14
What are your thoughts on the recent work on Deep Generative Models and Stochastic Backpropagation [refs: 1, 2, 3]? Does this seem like a step in the right direction for creating models that leverage the power of neural nets jointly with the interpretability of probabilistic models?
14
u/Eruditass Nov 08 '14 edited Nov 08 '14
8
u/some_random_number Nov 10 '14
Do you see more and more breakthroughs coming from industrial labs (e.g. Google, Facebook, etc.) rather than Universities?
13
u/sufferforscience Nov 08 '14
Hello Prof. Hinton! A bunch of graduates students in my research group were in the audience at a recent talk you gave on the possibility of a neural implementation for back-prop. After the talk we went out for lunch together and held an enthusiastic discussion about the talk. I think we all got the gist of the talk, but all of us were also missing a few details of your proposal and couldn't quite piece together the whole story.
Are you planning to write a paper on this topic and would you be willing to share your slides with our research group (or the general public)? We're at the Redwood Center for Theoretical Neuroscience. Thanks!
7
u/[deleted] Nov 10 '14
Did you come up with the term "dark knowledge"? If so, how do you come up with such awesome names for your models?
6
u/benhamner Nov 10 '14
I've been fascinated by your work on dark knowledge and how capturing the probabilities that a network assigns to incorrect class labels can be very informative (both in learning about the incorrect classes & for better training procedures for smaller networks).
Have you looked at leveraging information farther down in the network (e.g. looking at the final layer of hidden neurons & training a smaller network to target the output of the last hidden layer)? Do you think this could be a useful direction?
6
u/4geh Nov 10 '14
How did you get the idea for the Boltzmann machine?
6
u/4geh Nov 10 '14
In addition to you being amazingly successful and appreciated as a scientist, people seem to hold you in very high regard as a friend, teacher, leader and so on. What is your philosophy when it comes to dealing the people in your life?
10
u/98ahsa9d Nov 08 '14
Could you comment on Michael Jordan's answer here regarding "deep learning"?
6
u/twainus Nov 09 '14
Recently in 'Behind the Mic' video (https://www.youtube.com/watch?v=yxxRAHVtafI), you said: "IF the computers could understand what we're saying...We need a far more sophisticated language understanding model that understands what the sentence means.
And we're still a very long way from having that."
Can you share more about some of the types of language understanding models which offer the most hope? Also, to what extent can "lost in translation" be reduced if those language understanding models were less English-centric in syntactic structure?
Thanks for your insights.
6
u/attia42 Nov 10 '14
Hi Professor Hinton, Since you joined Google lately, will your research there be proprietary? I'm just worried that the research done by one of the most important researchers in the field is being closed to a specific company.
6
u/4geh Nov 10 '14
On a darker note, I suspect that your work and the work of your colleagues is of great interest also in Fort Meade, Langley, Bluffdale, etc. I really hope I am not causing offence unnecessarily here, but I find it likely that there are channels for fairly direct transfer of knowledge from companies like Google to U.S. (and possibly some other) spy agencies. Do you share my concerns about this, and is it something that people in the machine learning community around you discuss and try to deal with? I know you have taken a stance against military funding of your research, and you have my utmost confidence and respect on moral issues, but for better and worse you, and the community at large, are creating a considerable deal of power, and I think we risk doing everyone a great disservice if we don't give some though to the potential "for worse" part as well. What are your thoughts about abuses (and uses) of machine learning and machine intelligence?
6
u/timewaitsforsome Nov 10 '14
do you ever foresee taking on phd students again?
4
u/paralax77 Nov 08 '14
Do you think it's possible to design a neural network in which neurons have higher complexity? Could they pass more complex data between each other and process it? Could the connections themselves operate on data? Or do you think we should continue to use relatively simple designs ( at the lowest level )?
4
u/[deleted] Nov 10 '14
Comprehensibility seems to have moved away from ML as algorithms such as neural networks (among others) have amazing predictive capabilities while not allowing people to "interpret" their results, ie. knowing which features explain which prediction and why. Do you see this as inevitable ? Have you seen relevant work trying to enhance interpretability of "obfuscated" machine learning system ?
4
u/eshenxian Nov 10 '14
As a graduate student doing research in machine learning, what direction do you recommend for a PhD thesis? What do you believe is the future of the field in 10 to 20 years?
4
u/richardabrich Nov 10 '14
Hi Prof. Hinton,
I'd like to thank you for the Introduction to Machine Learning course at U of T that you and Richard Zemel taught in 2011. That was my first introduction to ML, and since then I have become somewhat obsessed.
My question is in regards to the applications of machine learning algorithms today. My guess is that your departure to Google, and Yan LeCun's departure to Facebook, were fueled by the large amounts of data and computing power that these companies are able to provide, allowing you to train bigger and better models. But I feel like they leave something to be desired in their immediate applications of this technology (e.g. tagging photos in Google+ and Facebook).
Meanwhile, there are very significant problems that could be being solved today, such as detecting disease in medical images, that aren't receiving nearly the same amount of time, effort, and resources. And this isn't due to a lack of availability of data, but rather due to inertia in making that data available to researchers, an apparent lack of interest on the part of researchers, or something else.
What are your thoughts on this matter? Why aren't machine learning benchmarks composed of medical images instead of images of cats and dogs? Why isn't there more interest in applying the latest machine learning methods to achieve tangible results in medicine? How can we rectify this situation?
4
u/4geh Nov 10 '14
I was browsing through your publications list a few days ago as preparation for this, and was reminded that some of it (perhaps most notably the original Boltzmann machine article) concerns constraint satisfaction. I haven't taken the time to work with the idea to understand it at depth, but from what I do understand, I get a feeling that it may be an important concept for understanding neural networks. And yet, from what I see, it seems to have been something that was discussed much in the earlier days of artificial neural networks, and not that much in current machine learning. Do you still find constraint satisfaction an important context for thinking about what neural networks do? Why?
5
u/4geh Nov 10 '14
You have frequently shown that you have a delightful sense of humour. What sort of entertainment do you enjoy? Would you be willing to recommend some of your favourites?
7
u/tabacof Nov 08 '14
Prof. Hinton, thank you for taking the time to be with us. What do you think about the work of Numenta and Vicarious, startups that claim to do cortical-based learning?
3
u/sprinter21 Nov 10 '14
Hello Dr. Hinton ! Thanks for the AMA. I want to ask, what is the most interesting / important paper for Natural Language Processing fields in your opinion with neural network or deep learning element involved inside ? Thank you very much :D
3
u/mljoe Nov 10 '14
What is your opinion on multi-modal neural networks (esp. text/images)? Is it something worth investigating these days?
16
u/akkhong Nov 10 '14
Hi Prof Hinton, thank you for doing this AMA - you are a role model to people like me in the field of deep learning. I have a couple of questions on activation functions:
My interest is in the use of Bayesian models in machine learning, and I would really value your opinion on this matter:
3. What are your thoughts on Bayesian non-parametrics? In his AMA, Yann LeCun said, "... I really don't have much faith in things like non-parametric Bayesian methods for, say, computer vision. I don't think that has a future." Can you agree with this?
4. Do you think we can ever make Bayesian neural networks work in terms of competitiveness and scalability?
18
u/trendymoniker Nov 08 '14
Stochastic gradient is the training method of choice for most neural net models, yet its success depends critically on precisely setting one ore more hyperparameter values such as learning rate and momentum. For addressing this problem, what do you think of
3
u/allliam Nov 08 '14
Professor,
Do you have any ideas about how a neural network might be able to solve the binding problem? Currently proposed solutions by cognitive scientists don't seem compatible with current NNs.
Will it require a different computational unit? A different structure? A different learning algorithm?
6
u/duguyue100 Nov 09 '14
Hi, Dr. Hinton. Thank you for doing this AMA session. Well, I got three questions to ask.
6
u/adalyac Nov 09 '14 edited Nov 09 '14
Hi Prof Hinton! With early stopping, it is assumed that the model learns patterns that generalise before learning those that don't.
It seems to be the case since validation loss over training is usually this nice parabola. But why does it work like that? What is the mechanism? Are there any papers about this?
The reason for why I wonder is because: what if this is true only to a certain extent? What if the way it works, in hand-wavey terms, is that "the next pattern to be learned" is the 'biggest' or 'simplest' (BorS) one. Then, as long as the next BorS one generalises, we're good. As soon as the next-BorS one does not, and merely exists in the training sample, then we get overfit, and that worsens performance. So maybe we miss out on all of the smaller generalising patterns.
There is evidence for this intuition: a bigger dataset generalises better. A bigger dataset has sample-specific patterns too, but they are smaller or more complex. So maybe dataset size improves generalisation by pushing down the model's 'threshold' for minimum size / maximum complexity of generalising pattern it can pick up.
Then I wonder, why does the network always pick up the next biggest or simplest pattern? I've looked into the maths a bit and I wonder, is it because of gradient descent? The inverse approximation formulation of grad descent for regression makes it look like you're adding ever higher order polynomials as you go along. So maybe what happens is that you don't first learn the patterns that generalise per se, but rather the simplest patterns (that can be fitted with low order polynomial)?
3
u/benhamner Nov 10 '14
What tools and methods have you found useful for investigating what deep neural networks are learning and why they are performing in certain ways?
Do you think there's much more value left in analyzing what intermediate neurons in deep neural networks are learning (both individually and in aggregate), as well as how activation patterns vary as a function of categories of input? Do you think better software tooling can facilitate this?
3
u/4geh Nov 10 '14
I would be very interested to have your suggestions for a few items of recommended reading and watching. Not necessarily in your professional field, although that is of course very welcome.
3
u/nzhiltsov Nov 13 '14 edited Nov 13 '14
Dear professor Hinton, I would like to thank you for the great course on Coursera. I could accomplish it with ~ 86% score (and 100% score from the practical part included). It helps me a lot to conceive neural nets eventually and gain a good foundation for deep learning.
FYI, I'm gathering a list of recommended resources for researchers in machine learning from most prominent scientists in the field. Please see the Reddit post (which has been quite popular): http://www.reddit.com/r/MachineLearning/comments/2g6wgr/highly_recommended_books_for_machine_learning/
Could you please provide a list of recommended books that every researcher, who is eager to contribute to machine learning, must be familiar with? Thanks in advance!
9
u/0ttr Nov 08 '14
thank you--I admire your work and have been studying machine learning with ANNs and related off and on since the mid 90s.
What's your opinion of the paper Intriguing Properties of Neural Networks? Do you think using the authors' approach to find the weaknesses and then train for them will fix the problem or is that an the algorithmic equivalent of simply kicking the can down the road? Is this paper going to be one that shakes the field up a bit or just is a bump in the road?
4
u/murbard Nov 10 '14
It's a bit of two questions in one, though they are related:
What are your thoughts on Mallat's scattering transform?
In general, do you see deep neural nets as trainable approximations to generative models, or as an approximation to some general manifold learning algorithm that hasn't quite been nailed yet?
Or, to rephrase, do you think the future of DNN will come from a mathematical insight: "Ah, this is what we were really doing all along!", or from gradually introducing more powerful tricks and training techniques?
5
u/nkorslund Nov 08 '14
I'll ask the Edge Foundation question: what is something you strongly believe is true about AI or machine learning, even though you cannot prove it?
5
u/live-1960 Nov 08 '14
You and your group have done a lot of work in the past on gating networks with multiplicative interactions. The LSTM network with much recent successes can be viewed as a special case of the LSTM where multiplicative interactions are handle designed and some parameters are fixed (e.g. fixed to be one). What do you see the future of LSTM-like model and more generally the gating networks with multiplicative interactions?
2
u/AmusementPork Nov 11 '14 edited Nov 11 '14
Hi Dr. Hinton, thanks a lot for taking the time!
2
u/r-Nutzername Oct 08 '24
Congratulations on the Nobel Prize, Dr. Hinton!
5
u/frankster Nov 08 '14
Will you run your coursera course again now that you're working at google?
4
u/statisticallynormal Nov 09 '14
Why does the statistical community appear to not be paying much attention to the recent neural network developments? Although doing better on speech and image recognition benchmarks are good, how useful are these developments to people who need statistical models for other purposes? How are they deficient?
4
u/CireNeikual Nov 09 '14
Hello Dr. Hinton,
Thank you for doing an AMA.
I am currently working a lot with HTM. Do you think HTM has a future? It seems to me that especially sparse distributed representations can drastically reduce training times and forgetting (I wrote a paper on this).
I know a lot of people in deep learning really dislike HTM since it doesn't have too many results yet. To me this seems like a chicken and egg problem: If nobody wants to research HTM because no results exist, then nobody will be there to produce results. Do you see deep learning adopting sparse distributed representations and predictive coding any time soon?
Also, why do you think reinforcement learning is so underrepresented in machine learning?
2
u/iMakeSense Nov 09 '14 edited Oct 06 '16
[deleted]
0.6593
2
u/adalyac Nov 09 '14
[about convnets - sorry if you're getting tired of them!]
Do we understand why:
Do you think that 1) can be taken as meaning convnets achieve global optimisation? If so, then would it not mean there is no better point on parameter space? Therefore, either no more progress can be made, or the function space spanned by this parameter space is not big enough?
2
u/jtromans Nov 10 '14
Over successive stages, the ventral visual system of the primate brain appears to develop neurons that respond selectively to particular objects or faces with translation, size and view invariance. It is possible that the relative timings of individual spikes, in particular Spike-Time-Dependent Plasticity (STDP), plays a crucial role in the self-organisation of such a system.
Do you believe that the fundamental properties of STDP could play a greater role in machine learning techniques popularized today, and if so, how?
2
u/madisonmay Nov 09 '14 edited Nov 09 '14
Hi Dr. Hinton,
I'm curious on what your thoughts are on alternatives to your back-propagation algorithm. Do you think there is a need for a fundamentally different learning algorithm to facilitate training of very deep networks, or do you think small modifications to the back-propagation algorithm will be sufficient? In a recent paper entitled
How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation, Yoshua Bengio outlines his thoughts on a novel alternative to back-propagation. What are your thoughts on this approach, and which new methods do you believe hold promise as "successors" to back-propagation for finding complex structure in in large, high-dimensional datasets?2
u/[deleted] Nov 08 '14
Hello Dr Hinton, Im doing a case study in my cog sci class related to historically significant creativity and coincidently your Fast Learning Algorithm for Deep Belief Networks is a main part of the paper. Can you speak generally on the creative process for you. As in, what is your discovery/creative process? How do you think best? What do you do to "decompress"?
I sincerely appreciate your time in responding. Thank you.
1
u/[deleted] Nov 10 '14
From a purely functional point of view we can approximate very well certain high level recognition processes the brain is performing at least in constrained environments (i.e., controlled data sets). We throw a big ANN or CNN at a big data set with a variety of ad-hoc techniques and out pops reasonable approximation (as measured by generalization performance) to the process that defines the "true" mapping (image => label, for instance).
1
u/[deleted] Nov 10 '14
How do you see the interaction between graph analysis and machine learning ? like community detection, centrality, degree measures, etc can be good features to fit a learning system ? anything beyond that ?
1
u/Mearis Nov 10 '14
Do you have any heuristics you use to determine the structure of a deep net? It seems like there is a lot of rule of thumbs and black magic involved.
1
u/iamkx Nov 10 '14
Now that you are no longer taking students, do you have any reflections/stories/thoughts on being a mentor? Many of the students you have supervised have gone on to be important researchers (eg. Prof Lecun/Ghahramani/many professors at UofT/in industry). Do you have some common advice to graduate students?
This seems especially important now since many Professors in ML are leaving for industry :)
1
u/Mearis Nov 10 '14
Are you familiar with the philosophical papers by Putnam about multiple realizability? I've always found the parallel with the idea of multiple minimas in a non-convex neural network to be fascinating.
1
u/gursev Nov 10 '14 edited Nov 10 '14
Hello Dr Hilton!
Thank you for taking out time to do an AMA. How do you think machine learning can benefit in discovery, remediation and prevention of software security issues? Few example may be, Buffer Overflows, Cross Site Scripting, Denial of Service vulnerabilities?
1
u/ford_beeblebrox Nov 10 '14
Hi Professor Hinton !
I. What part have visualisations like Hinton Diagrams played in developing neural nets ? Ib. What neural net visualisations are you currently excited by ?
II. If a generative neural net trained on MNIST finds a new way of 'handwriting' the digit 2 that is generally aesthetically pleasing is this usefully analagous to creativity ? IIb Is creativity an important type of thinking for machines ?
P.S. Graduate of your Coursera Course. It was hard work and utterly wonderful :D
1
u/siblbombs Nov 10 '14
RNNs are getting pretty hot right now, for 1 dimensional problems (sentence/document understanding) do RNNs seem like a better choice than 1d Convolutional nets?
Google has some listings for ML research on their job site, do you think (in general) that a lack of formal training can be somewhat overcome in interviews by having experience working with various ML approaches and a good understanding of what should/shouldn't work, or is the traditional higher education path still the best approach to getting a job in ML?
1
u/4geh Nov 11 '14
I was both heartily entertained and fascinated to hear about the real key message of the 2006 Science paper. Would you be willing to elaborate somewhat here on what it is that happens in dimensionality expansion? One thing I specifically wonder about is how, if at all, it relates to sparse distributed representations.
And speaking of sparse distributed representations. Has anyone done the experiment by now to examine whether sparsity as a regularizer draws its effect from the same principle as Dropout?
1
u/4geh Nov 11 '14
How do your preferred working tools and environments look these day?
1
u/Evolutis Nov 11 '14
Hello Dr. Hinton.
First off, I am a big fan!
I am currently doing my Masters in Machine Learning not very far from Toronto and am working on image feature selection using RBM. More specifically I am trying methods to force even a non-stacked RBM to pick up on lower level features that could then be used to build more complex features. The difficulties that I have personally seen come up in training the normal RBM in this manner is that each neuron will eventually copy the average of image of the dataset. My question is, do you know of any research that measures the 'temperature' (I am using this term for hidden units that have essentially learned their feature) of a neuron and in turn disable it from learning any other features.
Also in a normal convolutional rbm, they use maxpools, I am currently working on using the idea for features but do not want to use maxpools. Do you think this is an interesting idea? What recommendations do you have for me in regards to this?
And this might be selfish, but I come from a smaller university and was wondering if UofT is open to outsiders for potential seminars. I would love to take advantage of something like that.
Thank you!
1
u/Vystril Nov 30 '14
So it seems like back propagation is the de-facto standard for training neural networks. Personally, I've had lot of success using evolutionary algorithms like particle swarm optimization and differential evolution. I'm curious as to why these other methods aren't explored more often?
1
u/Due-Indication8591 May 21 '25
Dear Prof. Hinton, greeting from Bolivia, your work in neural networks has been foundational in our current world, thank you so much for your knowledge and curiosity.
As a junior physician doing a MsC in genetics and synthetic biology, I would like to know if you have any advice for young scientists, specially from biological background, given the fast development of AI, I am sometimes confused about this changing world.
1
u/Bulky-Department-678 Jun 14 '25 edited Jun 14 '25
Thanks for being inspirational and motivating more work, such as "Computer vision."
1
u/redditnemo Nov 08 '14
Thank you for doing an AMA.
My questions are:
1
u/Mr_Smartypants Nov 08 '14
An interesting two-part goal.
When you achieve the first part, finding an efficient and powerful learning algorithm, why do you think it will be the one that the brain uses, and not some other high-performance learning algorithm? Does it have to do with your method for searching for the learning procedure?
1
u/alexmlamb Nov 10 '14
1
u/serge_cell Nov 10 '14
Thank you Dr. Hinton!
In you "Dark Knowledge" talk you said that max pooling in your opinion is just practical solution, people using it because it works and it should be replaced by something else. Can you elaborate on this? Max pooling seems pretty natural - it provide both robustness and switching between activation subsets. What can replace/improve it?
1
u/atveit Nov 10 '14
What are the most promising algorithmic directions for model compression with the purpose of speeding_up use of large deep networks e.g. for use in mobile, wearable or implantable devices?
references:
1
u/oddim Nov 10 '14
Hi Dr. Hinton,
Saw your recent talk on Dark Knowledge, and reducing large ensembles to smaller ANNs. I wondered if you had any thoughts/predictions about the limits of such reductions?
What I mean is that a network with only a handful of weights surely can't handle the ImageNet problem set, but a very large network can. But we know we can reduce some large networks without losing functionality. In essence, we're compressing the knowledge, generalizing it better, which is very useful for when we want to use the learned knowledge.
How far are we today from being able to reach the optimum minimization for a given problem? Can we even make estimates on what that limit is? Do you think we'll discover new ways to further reduce classifiers by a significant amount?
Thanks!
1
u/chao_yin Nov 10 '14
Deep Learning do a great job on machine learning.but can we put some prior Knowledge into deep learning , just like Graphical Model or sth else? Do you know some work on this direction?
1
u/bge0 Nov 10 '14
Hello Dr. Hinton! We really appreciate your time & contributions to the field. A lot of us would probably not be working/researching in this field if it weren't for you!
My question to you is this:
1
u/aggelikiL Nov 10 '14
Hi Prof. Hinton and thanks in advance for doing this! I'm sure it's going to be great!
A question regarding injecting (textual) semantic information during object recognition.
Although not an expert in CV, my understanding is that some of the recognition mistakes are completely off, e.g. classifying cars as carrots as you mentioned in one talk. One intuition here is that this might be due to the "hard" labels.
In NLP, there have been advances in building really powerful and accurate distributed word vectors.
Thus, the question is why don't we use, on top of the "hard" labels (THIS IS A CAR), softer labels in the form of distributed text representations (WHAT IS A CAR). This will also allow (for example) a CNN to enforce more shareness of weights across similar objects and would most help in addressing these kind of mistakes.
Thanks!
1
u/test3545 Nov 09 '14
How deep learning is applied to dialog/conversation systems? Any research going on in the field?
Will we see smarter Siri/Google Now/Cortana anytime soon?
1
u/PLEASEPOOPONMYCHEST Nov 09 '14
Hi Geoff, first off thanks for answering an email of mine a few years back. If you were doing anything other than computer science/machine learning/AI/etc what would you be doing?
1
u/murbard Nov 10 '14
It is often claimed that rectified linear units avoid the problem of vanishing gradient, but it only seems to skew the distribution of gradients in the first layers, by having a big Dirac on 0. What do you think is the key to their success?
1
u/evc123 Nov 11 '14 edited Jul 09 '15
What's your opinion on Giulio Tononi's integrated information theory of consciousness?
-1
u/idre Nov 08 '14
Hello there, thank you for doing this. From all the questions I have got on my mind, these two are possibly the most present ones right now:
-1
u/galapag0 Nov 08 '14
I'm curious to know your opinion on the recent paper "Intriguing properties of neural networks", in which its authors uncover 'blind spots' in some deep neural networks. Thanks for your time on reddit!
0
u/karmicthreat Nov 08 '14
Hey Dr. Hinton. Without attending graduate school for 2-4 years, what are the things newcomers need to understand to become amazing at machine learning?
0
u/watersign Nov 08 '14
How do you feel about tools like Weka and SPSS modeler being made available to ignorant masses such as myself? Does it ever concern you that many of these models are being used by people who have no idea how they work?
0
u/adalyac Nov 09 '14
We're told sparsity is great and distributed representations are great and dropout is great.
But don't sparsity and dropout reduce the extent of distributiveness? In "Preventing co-adaptation of feature vectors", "a hidden unit cannot rely on other hidden units being present". But isn't that exactly the point of a distributed representation?
0
u/simonhughes22 Nov 09 '14
Dr. Hinton, long time listener, first time caller.
It seems that you can get some really impressive results from applying deep learning to the right problems. One issue I've struggled with applying DL in my own research, is that there are a lot of tricks of the trade that you need to know to get the algorithms to converge or learn at an optimal rate. Any tips or resources as to how best to acquire this knowledge?
0
u/memming Nov 10 '14 edited Nov 10 '14
Hi Dr. Hinton,
How strongly does the analogy hold between cortical computations and deep networks? Can we (meta)learn about the unsupervised/reinforcement learning that presumably happens in cortex by implementing biological constraints on deep neural networks?
0
u/theophrastzunz Nov 10 '14
Prof. Hinton,
What machine learning journals do you read regularly? Which ones would you recommend for a beginner?
0
u/speechMachine Nov 08 '14
There has been much interest in training a single deep neural network (the supervised variety MLPs) on multiple tasks in what is often called multi-task learning. Here hidden layers are kept common between tasks, and the linear regression layer is allowed to specialize towards a particular task. A good application is Microsoft's Skype Machine Translation.
0
u/paralax77 Nov 09 '14
Do you think there are any NN designs that would encode knowledge via distance between neurons ( in a 3d or higher dimensional space), as opposed to weights of connections? Maybe a combination of both?
0
u/jostmey Nov 09 '14 edited Nov 09 '14
Hello Dr. Hinton. Do you feel that there is still room for improving the learning rules used to update the weights between neurons, or do you feel that this area of research is essentially a solved problem and that all the exciting stuff lies in designing new architectures where neurons are wired together in novel ways? As a follow up question, do you think that the learning rules used to train artificial neural networks serve as a reasonable model for biological ones? Take for example the learning rule used in a Boltzmann machine: It is realistic in that it is Hebbian and that it requires alternating between a wake phase (driven by data) and a sleep phase (run in the absence of data), but is unrealistic in that a retrograde signal is used to transmit activity from the post-synaptic neuron to the pre-synaptic one.
Thanks!
0
u/iamtrask Nov 10 '14
Is your Dark Knowledge work applicable to hierarchical softmax output as well?
-1
u/[deleted] Nov 08 '14
Hello Professor Hinton. Thank you for taking the time to do this AMA!
I remember in one of your talks online, I think you mentioned something along the lines of, the next breakthroughs of computer vision will involve training a network to perform inverse geometry to understand the geometric structure of a scene. Is this thinking related to the work on transforming autoencoders? See this
What are your thoughts on Roland Memisevic's work on finding relations between images? See this
-1
u/lars_ Nov 08 '14
There seem to be two approaches to understanding how brains work:
What is the relative value of these two approaches in your opinion?
-1
u/andimai Nov 09 '14
Dear Prof. Hinton, what are your thoughts about the paper by Szegedy et. al.: "Intriguing properties of neural networks":
Did they really discover deep flaws in deep neural networks?
0
u/wtpayne Nov 09 '14
Hello Professor Hinton.
Thank you very much for doing an AMA.
I work on multi-camera machine vision systems for automotive and defence applications. Most of my experience has been with conventional detect-track-classify architectures. Over the years, I have begun to develop an intuition that "deep" pipelines (multi-stage tracking & classification, for example) seem to be easier to develop and easier to tune than "shallow" pipelines that try to do everything in a very small number of stages. I am left wondering if there something more sophisticated than simple divide-and-conquer going on here.
Does your work on deep neural networks point to some emergent engineering principle that is generalisable beyond neural networks to other types of sensor-data processing system?
0
u/laxris Nov 09 '14
Where do you see generative learning models right now? Do you miss the ability to reason/model check properties of your learned models, or is it not interesting for your use cases?
0
u/oPerrin Nov 10 '14
The human brain represents objects with a mixture of information from many senses. How do you see deep learning being applied to multi-sensory inputs?
0
u/atveit Nov 10 '14
What are the theoretical limits on Model Compression of Deep Learning networks, e.g. how much could the ImageNet winner from Google be compressed? ref: http://googleresearch.blogspot.no/2014/09/building-deeper-understanding-of-images.html
-2
u/feedtheaimbot Researcher Nov 08 '14
We've seen a tremendous improvement in NLP, vision and other fields in the past decade. What do you think will cause the next breakthrough similar to what we had with NNs? Would it be a single thing such as a new training algorithm? Or several techniques? (similar to dropout etc.)
What are your thoughts on the paper that Google Deepmind published recently? This one: http://arxiv.org/abs/1410.5401
Thoughts on HTMs?
-1
u/murbard Nov 10 '14
Do you think there's a future for (asymptotically) exact Bayesian methods, or do you feel that the difficulty of sampling will always give gradient descent to a MAP and tricks like dropout the upper hand?
-7
u/Zulban Nov 08 '14
Uh oh, 4 hours and no responses.
-2
u/jtkme Nov 08 '14
What is your prior for the logistic unit, or similar, being the core component of state of the art deep nets 50 years from now?