Tensorflow 2 Internals – Lessons learned from creating a 50 hours course

Reading Time: 10 minutes

I was asked to teach a course about a new version deep learning framework by google – Tensorflow version 2 to a company of highly technical people creating smart acoustic and microphone systems for recording studios. Tensorflow 2 came out during the middle of 2019 and is very different than the previous version, so there was a lot of learning to do and trial and error in the face of unknown code, api and bugs.This took me about 3 months of almost full time job to complete. After the course finished I also asked the students to fill in a review about it so I will have their input. During the process I learned a lot about learning, creative work and presentation. I am sharing this here both for my own sake and for anyone else who might benefit from this knowledge.

How to learn new material

1. Map information sources
   1. Go online and filter and map all quality sources of material. When the subject is quite new, there are not so many quality information sources online. Doing a first map of all the good quality sources you found gives access to information and at the same time confidence that you are not going to miss out on information because you didn’t know of another source. For example these sources can be:
      1. Youtube videos
      2. Official documentation
      3. Open source side projects
      4. Books
      5. Online lectures
      6. Conferences\events which have a written online record of what was taught
      7. Stack overflow questions\comments
   2. There are sources of sources online – pages that reference to other quality information sources, a sort of dynamic online index. These can be reddit posts, youtube lectures with references (by the google tensorflow team in my case), good github pages and more. Go over all of them and map the most relevant and good information sources you will use.
   3. Also possible to ask friends for reference materials – in my case, the subject was so new, almost non of my tech friends could help
   4. While you do this first initial mapping, you will inevitably encounter the most significant topics of to learn (and teach), start writing down recurring topic names\chapter or topics which you saw might be extra interesting or relevant even though they are not the main topics.
   5. You will also encounter knowledgeable people or specific quality sources, check all their content, you might find more missing gems there. 
   6. Mark specific references which you might use in whole or part while teaching – either as teaching material or exercise material.
   7. In my tensorflow learning I found these resources to be extremely helpful
      1. Google TensorFlow guides and tutorials
      2. Everything by Aurélien Géronhttps://github.com/ageron/tf2_courseHis book: https://learning.oreilly.com/library/view/hands-on-machine-learning/9781492032632/
      3. Stanford class https://web.stanford.edu/class/cs20si/
      4. Tensorflow Youtube channel, especially with the “Inside Tensorflow” playlist – https://www.youtube.com/watch?v=yTJ8QydIgVQ&list=PLQY2H8rRoyvzIuB8rZXs7pfyjiSUs8Vza and videos from TF conferences.
      5. These reddit posts
         1. https://www.reddit.com/r/MachineLearning/comments/7u1hki/d_recommendations_for_tutorials_on_tensorflow/
         2. https://www.reddit.com/r/tensorflow/comments/bliecp/best_way_to_learn_tensorflow_20/
      6. This github page with a huge amount of resources and references https://github.com/Amin-Tgz/Awesome-TensorFlow-2/blob/master/README.md
2. Filtering
   1. Briefly check each information source to see if it is good or just someone posting content more for self promotion\marketing and less for giving real value and insight info about a topic
   2. make sure you only have the best reference for each sub-topic, If a subtopic has more than 3 information sources, save all of them, use only the best\main 3 and if you need extra info then you have where to look.
   3. While you are filtering you will start to learn the basic concepts of what you are learning, you will start to get a feel for what you should focus more on later, what is extra interesting.
   4. Going over tensorflow materials I found that a lot of the materials might say they are for tensorflow 2, but actually they are for tensorflow 1. Also, a lot of the materials were brief and shallow explanations without real proper content and value. This filtering step took me about two days to finish.
      
3. First reading
   1. White papers\Architecture reviews – read thoroughly. I started off my tensorflow learning process by reading the google whitepaper from 2015 for the framework. Interestingly, I also finished learning tensorflow with this white paper, after I went over all the other material and presentations. When I started learning the white paper helped me understand which important/interesting/complicated topics I will encounter and what I should teach. I ended with the white paper because it had in depth explanation about the specifics of the framework which were hard to find in other places.
   2. Official documentation – skim. The purpose is:
      1. Check what will be hard and tricky to learn (which means it will consume a lot of time and effort) and what will be easy
      2. Check what are the most important things you should focus on first, what are the basics of the main subject which all other sub topics depend on
   3. Do at least one significant, non trivial tutorial, to start “feeling” the topic better and raising more questions.
      
4. In-depth reading
   1. Once you have listed in order the important things to focus one, start going over them one by one, using the references which you found.
   2. This stage can be unexpectedly long and hard, but it is important for your overall familiarity of what you are learning and for your confidence in what you learned. Do not skip and\or rush it.
   3. Write down short summaries and the most significant sub-topics you encounter, along with links to the reference materials where you learned of these sub-topics. This will be the basis of what you will be teaching later.
   4. If you find a specific sub-topic is taking you longer then expected. Or, maybe the sources you find don’t cover it well. This is a good sign. It means you have found an area where you can learn something new not many people online know and you can teach it later.
      1. If it is an immediately important sub-topic –  Focus on it, understand it, if required read the open source until you familiarize yourself with the topic. Take notes during your learning, this will be your source of information (and your audience’s).
      2. If this is not immediately important – write this subtopic down in your open items and make sure you have a task to come back and research this subtopic later
   5. While you are learning, you will encounter questions and things you don’t understand. If it is part of your learning flow, go ahead and explore them. If not, write the open questions you have in your notes and continue learning the main thing you set out to learn. These questions are important. Your real understanding comes from asking these questions (which means you understand what you are learning and you try to transfer it to something new that wasn’t completely explained). Also, when later you will answer these question the search for the answer and the final answer will greatly enlarge your overall understanding and confidence in your own understanding.
   6. Check back on previously opened questions which might be related to a subtopic you are now learning, Most chances you will be able to answer it. If you see you are still stuck even after the depth reading this might be related to point 4 above, or you might need to learn another sub-topic before you could answer the question.
   7. Practice and recall what you learned – Try out things by yourself (your own code) based on what you learned, see if they work or not. Try out special use-cases or or esoteric implementations which you thought of but are not written in your references – this will both help you remember the subject better and will raise issues you are not sure about in your knowledge. Also, this could be good class examples or exercises later.
   8. While you are learning, if you encounter open questions people posted which you find answers to – post the answers online, it will help your own understanding, will help others and will promote you.
      
5. Points 1-3 can be executed simultaneously, interleaving them until you cover all the interesting references. Point 4 needs to be handled by itself in a focused manner.
6. Summarize, log or write down things you are not sure of. Later when you learn more deeply you can delete these summaries or change them according to your new found knowledge. You can mark things you are unsure of (I use triple question marks – ???). Also, writing it down allows you to think it through, understand better and remember what you learned (even if partially). Don’t be afraid to “waste” your time writing down something you might delete later.
7. Bugs in the framework can be very frustrating and time consuming. When a code is not working as expected, make sure it is not working first by minimizing it as much as you can and checking the documentation if it should be working. Then, search google and github to see if anyone else wrote anything about it. If there is nothing written, you will need to double check yourself to make sure it is a bug.
8. Most of my personal tensorflow learning was through encountering bugs when I was trying to run my own code, from having the class ask me in-depth questions and from just experimenting with different graph structures and different ways to write similar functionalities. For example, everything I was running in eager mode I tried also running in graph mode and vice versa. One of the trickiest part to learn was to try and understand what goes under the hood with tf.data (I used the Inside Tensorflow youtube video about it to great lengths – https://www.youtube.com/watch?v=kVEOCfBy9uY&list=PLQY2H8rRoyvzIuB8rZXs7pfyjiSUs8Vza&index=6). One of the things that wasted a lot of my time was a bug with TF2.1 (which was released while my course was running) and TensorBoard – it wasn’t profiling GPU usage. Took me a full day to accept that this is a bug and not something wrong which I am doing and an extra 1 day (and 300$ cloud server charges) to setup a working environment with TF2.0 instead of TF2.1.

What I learned about preparing presentations for each topic

The cool thing is that if the learning part was done right, this next part is straightforward. You already have most of the materials you want to talk about and present. You also have references to interesting things to show in class and you have a breakdown of the information dependencies between the different subtopics. You also already went through many presentation on the same topics so you can follow their lines if you saw the presentations your good.Along with powerpoint presentations I use jupyter notebook to run python code in class. The jupyter notebook allows to show how code runs in real time, when exceptions occur and why and also allows an interactive way of teaching in which you can challenge the class with questions about how specific code will run and see the results live.
Guidelines:

1. Put all your summaries and reference materials infront of you in a notebook or in powerpoint
2. Add any other points you think are relevant for the presentation
3. At this stage you will see how about how long a presentation for one topic will take
4. Separate all the material to
   1. Presentation – Powerpoint\slides
   2. Presentation – colab\jupyter notebook code
   3. exercise
   4. Bonus information\practice
5. Fully create the presentation slides 
   1. Write notes to yourself which you will use in class
   2. Especially write notes if you are going to present something that connect to something later on in the presentation
   3. Explain every keyword you talk about
6. Fully create jupyter code notebooks
   1. Try to keep text to a minimum, but enough of it so that you will remember what to talk about
   2. Make sure all dependencies are installed as part of the notebook execution flow – good both for teaching about dependencies and making sure your code will run on any machine.
   3. Separate the topic into sections – help organize thoughts and manage time while presenting (maybe you will want to skip some sections if you run out of time)
   4. Along with the jupyter notebook, you should have a small reference notebooks with things you should talk about in class while presenting the notebook.
7. Repeat 1-6 until you feel that you covered everything for the topic you wanted
8. Points 1-2 should be done separately to points 3-5. Points 1-2 require imagination, abstract thinking and experimentation, if you combine it with the technical details of creating and editing the presentation you will get frustrated and won’t be able to do an effective job.
9. While you build your presentation you might find out you are missing knowledge about a specific sub-topic, go back to the in-depth part above and research into that topic.
10. Create buffer material which you will teach if there is extra time. Your original time estimates will never be completely accurate. If you run out of time you can start off the next class from where you stopped. If you finish before time, the extra buffer material will help you fill that void.

What I learned about presenting

1. The focus should be on having people understand what you are teaching, not in covering all the materials. It is ok (and advised) to skip\miss parts if it means people will understand what you teach better
2. Take pauses when presenting, laugh with the audience, tell stories. Actively think in advance what stories you will during the presentation – it both fills time, gives rest and fun
3. Along with regular break times, add mini breaks during your presentation. In these mini breaks your talk or show something cool which relates to the subject being taught. This both helps to refresh and interest people in what you are teaching further.For example, I showed cool research videos of the latest things in deep learning and deep fake. Later in the reviews the students said this was a cool addition to the course.
4. If something interests you, it will interest the audience both because it is an interesting topic and because of your personal interest and excitement
5. About making perfect presentation, error-less, interesting and smooth – The audience is extremely focused in trying to grasp and understand new concepts. They don’t notice the imperfections in your presentation, they notice the imperfections in their understanding of the material. Don’t worry about making it perfect, worry about making it and making it interesting for you while putting as much effort as you can in bringing the audience value.
6. In my course, I thought I did a poor job with explaining everything. But, the students told me that the tools I gave them will be very valuable for their work, and the code samples were very interesting for them.

About time management in face of unknown materials or task

When you start working on it, there are many things you are not familiar with, many things that can suck up your time without you being prepared. some guidelines on how to manage that:

1. Briefly skim the material you need to learn\present beforehand to have a rough estimate on how complicated it is. This is why the first stage of mapping your data sources when learning is so important.
2. If there are many tutorials and explanations about a subject you can estimate time quite accurately. The risk is covering subjects which don’t have a lot of easily accessible information – these usually consume most of the time.
3. It is sometimes scary or demotivating to work only to find out you have much more work than expected. To handle that, keep a very elaborate task list with priorities and always focus on the small next step, just on what’s the most important next small thing. Start gaining small wins, these will help you get the motivation to cover an unexpectedly large topic.
4. If you catch yourself studying deeply into a subject that might not be so important, stop. Write it down to check later and address the next subject on the list. You might find a subject is not so important only after you dive into it and find out it is raw, buggy or not the main focus of what you are studying\teaching.
5. Always remember, you don’t have to know all the exact details of everything. You need to know enough to get by and enough to teach your class.
6. If you are stuck and can’t progress, a helpful tip is to write down a question for yourself: “why am I stuck?” and answer it with atleast 5 different answers. You will find out exactly what’s blocking your path and will be able to tackle it head on and “unstuck” yourself.

Archiving and references

1. You should use a notebook or other method of writing and archiving for :
   1. Summaries of important, or at first seemingly important, subtopics and subjects with reference to where you learned it
   2. Reference materials
   3. Ideas to explore\Open questions
   4. Tasks
2. I used 3 evernote notes
   1. “resources” – All the references I found and my initial raw ideas
   2. “tasks” – The next tasks I should handle and any open items\unclear things I want to learn
   3. lecture – The structure of what I will present, the subtopics, main concepts, exercises and bonus references for class