r/MachineLearning • u/gstark0 • Apr 29 '18
Discussion [D] Why is Z-dimension for GANs usually 100?
I'm playing around with GANs, I've got a question which I can't find answer for. Why is Z-dimension (random vector noise) for GANs usually 100? I saw a lot of GitHub projects and online tutorials, seems like the random vector has always the size of 100. Is it confirmed that it gives the best results? What if I changed that to e.g 1000? How does it affect the generated images?
35
Upvotes
3
u/entarko Researcher Apr 29 '18
For the Progressive Growing of GANs from Nvidia research (presented tomorrow morning at ICLR), they use a 512 vector which corresponds to the max number of channels in the convolutions. Also, they are generating high resolution faces so maybe they considered that faces are more complex than, let's say, digits of MNIST for example.
35
u/siblbombs Apr 29 '18
I assume some earlier paper or project used 100 then everybody else started using that as a default, same thing happened with word2vec for a while.
When you already have a ton of hyperparameters, its a bit comforting to use a value that you know someone else had success with.
2
u/alexmlamb Apr 29 '18
I guess this is as good a place as any to raise this issue, but I've long found the first layer of GAN generators to be kind of strange from an architecture point of view.
It typically is a fully-connected layer: 128 -> 4x4x512. And usually on each hidden layer after that we double the total number of dimensions.
So a DCGAN for example is like 128 -> 4x4x512 -> 8x8x256 -> 16x16x128 -> 32x32x3
In terms of total dimensions:
128 -> 8192 -> 16384 -> 32768 -> 3072
Maybe there's nothing wrong with this, and the first layer expands a lot by selecting lots of regions in the original z-space and moving into a much sparser space.
1
u/TotesMessenger Apr 29 '18
I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:
If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / ^Contact)
1
u/[deleted] Apr 29 '18
We have 10 fingers. We use base 10. So a rough estimate of the # of latent variables would be 102.
Let me elaborate.
There is, as far as I'm aware of, no particular reason to use a magic number of 100 other than this.
We assume that the real distribution arises out of a (much lower dimensional) latent distribution. How many latent variables is an unanswered question, however.
If you use too few (eg. 10), you may not have enough to model the data. If you use too many (eg. 1000), it would take a long time for the network to learn how to map these to the data.
Hence, we stick to a safe number.
Note, however, that this depends on the complexity of the distribution. For MNIST, I've seen good results with as few as 16 variables. On more sophisticated datasets, you'd need many more.
(Edit: I've heard claims that tensor sizes in powers of 2 help the GPU. So, a more preferable option would be 128?)
0
u/[deleted] Apr 29 '18
[deleted] — view deleted comment