TTK28 Modeling with neural networks

Teacher: Bjarne Grimstad

Deep neural networks are being widely deployed in the industry to solve complex tasks that conventional methods have struggled to solve. There are several reasons for this breakthrough: more data, more compute, and improved algorithms and software. A perhaps equally important factor to the success is the flexibility of neural networks as a modeling framework. Without explicit programming, neural networks can learn to solve complex tasks from data. In this paradigm, the job of the engineer has moved from programming models and logic, to designing meaningful machine learning tasks.

This specialization course gives an introduction to neural networks. We will see how neural networks, being computational graphs, can be used to build very generic hypothesis spaces of nonlinear models. With these models we can solve regression, classification or other types of learning tasks by simply selecting an appropriate cost function. At the end of the course we will look briefly into probabilistic neural networks and how to apply neural networks on small datasets. During the course we will present industrial applications where neural networks are used to model, understand and optimize processes.

The goal with the course is to add neural networks to your problem solving toolkit. After completing the course, you should: understand the basic concepts of neural networks; be able to develop and apply a neural network; understand the advantages and limitations of neural networks; be able to digest new ML research and continue developing your skillset by self-study.

• Lecture 1 (2 hours)
  • Course introduction and motivation – why learn from data?
    • Definition of a learning task (dataset, cost function, model) – how to learn from data?
  • Different types of learning tasks: supervised (classification and regression) vs unsupervised (clustering, etc.).
• Lecture 2 (2 hours)
  • Linear and logistic regression – maximum likelihood, cost functions
  • Nonlinear features (linear in parameters) vs. nonlinear regression (nonlinear in parameters)
• Lecture 3 (3 hours)
  • Neural networks – representation (layer composition, activation functions, popular architectures)
  • Neural networks – learning (Part 1): back-propagation
  • Neural networks – learning (Part 2): stochastic gradient descent
• Lecture 4 (3 hours)
  • Bias-variance trade-off (classic ML vs. modern ML)
  • Regularization methods (L1, L2, dropout, stochastic)
  • Model selection using cross-validation and hyper-parameter optimization (grid search, random search, Bayesian optimization)
• Lecture 5 (2 hours)
  • ML frameworks and implementation (PyTorch and TensorFlow)
  • ML in practice (advise on how to develop models)
• Lecture 6 (3 hours)
  • A statistical motivation: ML vs. MAP estimation
  • Probabilistic neural networks: a peep at the research frontier in ML
• Lecture 7 (1 hour)
  • Examples from the process industry
  • Guest lectures (2 hours)
  • Guest lecturers from the academia (TBD)
  • Guest lecturers from the industry (TBD)

NOTE: The date and location of the lectures is yet to be decided due to the circumstances of the Covid-19 outbreak.

Students will be given a voluntary project assignment or topic to study.

The exam will an oral presentation of the assignment. Date: TBD

• Course in matrix algebra and calculus (e.g. TMA4115 - Calculus 3, or a similar course)
• Introductory course in statistics (e.g. TMA4240 – Statistics, or a similar course)
• Introductory course in mathematical optimization (e.g. TTK4135 - Optimization and Control, or a similar course)

• Lecture notes
• Deep learning by Goodfellow et al. (freely available at https://www.deeplearningbook.org/)
• Material from the course Machine Learning offered by Stanford (freely available at https://www.coursera.org/learn/machine-learning)
• Selection of papers (TBD)