.. done ================================ Module 1 - Auto-Differentiation ================================ .. image:: figs/Autograd/backprop4.png :align: center This module shows how to build the first version of MiniTorch (mini-MiniTorch?) using only Scalar values. This covers key aspects of auto-differentiation: the key technique in the system. Then you will use your code to train a preliminary model. All starter code is available in https://github.com/minitorch/Module-1 . To begin, remember to activate your virtual environment first, and then clone your assignment: >>> git clone {{STUDENT_ASSIGNMENT1_URL}} >>> cd {{STUDENT_ASSIGNMENT_NAME}} >>> pip install -Ue . Module 1 is built upon the previous Module 0, so make sure to pull your files from Assignment 0 to your new repo. Please continue to follow the :doc:`contributing` guideline. .. toctree:: :maxdepth: 1 :glob: :caption: Guides derivative scalar chainrule backpropagate Tasks ****** Task 1.1: Numerical Derivatives ================================== .. note:: This task requires basic familiarity with derivatives. Be sure to review `differentiation rules `_ and the notation for derivatives. Then carefully read the Guide on :doc:`derivative`. .. todo:: Complete the following function in `minitorch/scalar.py` and pass tests marked as `task1_1`. .. autofunction:: minitorch.scalar.central_difference Task 1.2: Scalars ======================== .. note:: This task requires familiarity with the :class:`minitorch.Scalar` class. Be sure to first carefully read the Guide on :doc:`scalar` and to refresh your memory on `Python numerical overrides `_. Implement the overridden mathematical functions required for the :class:`minitorch.Scalar` class. Each of these requires wiring the internal Python operator to the correct :func:`minitorch.Function.forward` call. .. autofunction:: minitorch.scalar.ScalarFunction.forward Read the example ScalarFunctions that we have implemented for guidelines. You may find it useful to reuse the operators from Module 0. We have built a debugging tool for you to observe the workings of your expressions to see how the graph is built. You can run it in `project/show_expression.py`. You need to install pydot and networkx. You can alter the expression at the top of the file and then run the code to create a graph in `Visdom`:: def expression(): x = minitorch.Scalar(10, name="x") y = x + 10. y.name = "y" return y >>> python project/show_expression.py .. image:: expgraph.png :align: center .. todo:: Complete the following functions in `minitorch/scalar.py`. .. autofunction:: minitorch.scalar.Mul.forward .. autofunction:: minitorch.scalar.Inv.forward .. autofunction:: minitorch.scalar.Neg.forward .. autofunction:: minitorch.scalar.Sigmoid.forward .. autofunction:: minitorch.scalar.ReLU.forward .. autofunction:: minitorch.scalar.Exp.forward .. todo:: Complete the following function in `minitorch/scalar.py`, and pass tests marked as `task1_2`. See `Python numerical overrides `_ for the interface of these methods. All of these functions should return :class:`minitorch.Scalar` arguments. .. autofunction:: minitorch.Scalar.__lt__ .. autofunction:: minitorch.Scalar.__gt__ .. autofunction:: minitorch.Scalar.__sub__ .. autofunction:: minitorch.Scalar.__neg__ .. autofunction:: minitorch.Scalar.__add__ .. autofunction:: minitorch.Scalar.log .. autofunction:: minitorch.Scalar.exp .. autofunction:: minitorch.Scalar.sigmoid .. autofunction:: minitorch.Scalar.relu Task 1.3: Chain Rule ======================== .. note:: This task is quite tricky, so be sure you understand the chain rule, Variables, and Functions. Be sure to first read the Guide on :doc:`chainrule` very carefully and read the code for other ScalarFunctions. Implement the `chain_rule` function on FunctionBase for functions of arbitrary arguments. This function should be able to backward process a function by passing it in a context and :math:`d_{out}` and then collecting the local derivatives. It should then pair these with the right variables and return them. This function is also where we filter out constants that were used on the forward pass, but do not need derivatives. .. todo:: Complete the following function in `minitorch/autodiff.py`, and pass tests marked as `task1_3`. .. autofunction:: minitorch.FunctionBase.chain_rule Task 1.4: Backpropagation ========================== .. note:: Be sure to first read the Guide on :doc:`backpropagate` very carefully and read the code for other ScalarFunctions. Implement backpropagation. Each of these requires wiring the internal Python operator to the correct :func:`minitorch.Function.backward` call. .. autofunction:: minitorch.scalar.ScalarFunction.backward Read the example ScalarFunctions that we have implemented for guidelines. Feel free to also consult `differentiation rules `_ if you forget how these identities work. .. todo:: Complete the following functions in `minitorch/autodiff.py` and `minitorch/scalar.py`, and pass tests marked as `task1_4`. .. autofunction:: minitorch.backpropagate .. autofunction:: minitorch.scalar.Mul.backward .. autofunction:: minitorch.scalar.Inv.backward .. autofunction:: minitorch.scalar.Neg.backward .. autofunction:: minitorch.scalar.Sigmoid.backward .. autofunction:: minitorch.scalar.ReLU.backward .. autofunction:: minitorch.scalar.Exp.backward Task 1.5: Training ======================== If your code works, you should now be able to run the training script. Study the code in `project/run_scalar.py` carefully to understand what the neural network is doing. You will also need your Module code from the previous assignment as well. You can modify the dataset and the module with the parameters at the top of the file. Start with this simple config:: PTS = 50 DATASET = datasets.Simple(PTS, vis=True) HIDDEN = 2 RATE = 0.5 .. image:: simple.png You can then move up to something more complex, for instance:: PTS = 50 DATASET = datasets.Xor(PTS, vis=True) HIDDEN = 10 RATE = 0.5 If your code is successful, you should be able to fit the data like this: .. image:: complete.png .. todo:: Train a scalar model, and add the output training logs and final images like the ones above to your README file.