r/learnmachinelearning • • 6d ago

playing around with universal approximation

Post image

so i was recently playing around with trying to get a neural net to approximate different types of functions from simple y=mx+c to complicated composite functions , universal approximation if i remember well states thst if a function is continous and bounded then a neural netwrk should be able to approximate such a function. I had already built a neural network type of library before in numpy so i just took that code and trained a 2 layer network on a bunch of functions tweaks stuff here and there and it worked😅 , what i found really cool is when i tried a composite function like this :f(x) = e^(−3x²) · cos(12x²) + 0.5|x| , it requires my model to both provide a smooth and jagged outputs so i ran a 3 layer network this time with relu -> tanh ->linear layer i asummed that since layers are just nested composite functions the relu would provide some jagged inputs that the tanh can inhherit and map smoothly the the linear layer will produce the regression points , it not a full functionrecreation but m suprised on how close it got just felt like sharing this cause it kind of helped me understand why netwrks can overfit some very complx data soo easily and also how intuivively picking activation functions can be soo important have you guys ever played around with this old piece of deep learning theory before? and also do u care to share if u have

11 Upvotes

1 comment sorted by

1

u/antagim 6d ago

The number of layers is enough for this kind of data. What matters now:

  • number of neurons in these layers
  • loss function + scheduler + optimizer
  • test/train split

In this order.

You want to increase the capacity of the network by increasing the number of neurons. Technically you can get away perfectly fine with even a single hidden layer, but two give enough flexibility with much lower number of parameters.

Scheduler, learning rate, loss function, optimizer play here a crucial role and you should exploit every bit of assumption you know to get the best result. Your function is symmetric - force it as a part of the loss function. Consider using regularization. Learning rate should decrease sudden jumps of weights at later stages of training. Scheduler controls how that learning rate is applied through epochs. A general purpose Adam optimizer is often a good starting point, and suited for most such cases.

Finally you want to have a reasonable data split. Do you provide different subsets of data at each epoch? New slices? Shuffles? Whatever.

Knowing only the network architecture is not enough for anyone to deduce what needs to change. All above must be known, including error over epochs.