understanding black box predictions via influence functions

logistic regression p (y|x)=\sigma (y \theta^Tx) \sigma . Pang Wei Koh and Percy Liang. Simonyan, K., Vedaldi, A., and Zisserman, A. Here, we plot I up,loss against variants that are missing these terms and show that they are necessary for picking up the truly inuential training points. J. Lucas, S. Sun, R. Zemel, and R. Grosse. This could be because we explicitly build optimization into the architecture, as in MAML or Deep Equilibrium Models. Programming languages & software engineering, Programming languages and software engineering, Designing AI Systems with Steerable Long-Term Dynamics, Using platform models responsibly: Developer tools with human-AI partnership at the center, [ICSE'22] TOGA: A Neural Method for Test Oracle Generation, Characterizing and Predicting Engagement of Blind and Low-Vision People with an Audio-Based Navigation App [Pre-recorded CHI 2022 presentation], Provably correct, asymptotically efficient, higher-order reverse-mode automatic differentiation [video], Closing remarks: Empowering software developers and mathematicians with next-generation AI, Research talks: AI for software development, MDETR: Modulated Detection for End-to-End Multi-Modal Understanding, Introducing Retiarii: A deep learning exploratory-training framework on NNI, Platform for Situated Intelligence Workshop | Day 2. Which optimization techniques are useful at which batch sizes? In. Reference Understanding Black-box Predictions via Influence Functions reading both values from disk and calculating the influence base on them. Neural nets have achieved amazing results over the past decade in domains as broad as vision, speech, language understanding, medicine, robotics, and game playing. The reference implementation can be found here: link. Highly overparameterized models can behave very differently from more traditional underparameterized ones. Please try again. Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. can take significant amounts of disk space (100s of GBs) but with a fast SSD 2017. Then, it'll calculate all s_test values and save those to disk. https://dl.acm.org/doi/10.5555/3305381.3305576. In this paper, we use influence functions a classic technique from robust statistics to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. Alex Adam, Keiran Paster, and Jenny (Jingyi) Liu, 25% Colab notebook and paper presentation. Some of the ideas have been established decades ago (and perhaps forgotten by much of the community), and others are just beginning to be understood today. insignificant. , . In. The previous lecture treated stochasticity as a curse; this one treats it as a blessing. Thomas, W. and Cook, R. D. Assessing influence on predictions from generalized linear models. We'll consider the two most common techniques for bilevel optimization: implicit differentiation, and unrolling. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. In this paper, we use influence functions a classic technique from robust statistics to trace a . Proc 34th Int Conf on Machine Learning, p.1885-1894. Borys Bryndak, Sergio Casas, and Sean Segal. Model selection in kernel based regression using the influence function. Lage, E. Chen, J. Understanding black-box predictions via influence functions. The The datasets for the experiments can also be found at the Codalab link. This is a better choice if you want all the bells-and-whistles of a near-state-of-the-art model. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, Chris Zhang, Dami Choi, Anqi (Joyce) Yang. All information about attending virtual lectures, tutorials, and office hours will be sent to enrolled students through Quercus. ordered by harmfulness. How can we explain the predictions of a black-box model? The algorithm moves then International Conference on Machine Learning (ICML), 2017. Acknowledgements The authors of the conference paper 'Understanding Black-box Predictions via Influence Functions' Pang Wei Koh et al. It is known that in a high complexity class such as exponential time, one can convert worst-case hardness into average-case hardness. ICML 2017 best paperStanfordPang Wei KohPercy liang, x_{test} y_{test} label x_{test} , n z_1z_n z_i=(x_i,y_i) L(z,\theta) z \theta , \hat{\theta}=argmin_{\theta}\frac{1}{n}\Sigma_{i=1}^{n}L(z_i,\theta), z z \epsilon ERM, \hat{\theta}_{\epsilon,z}=argmin_{\theta}\frac{1}{n}\Sigma_{i=1}^{n}L(z_i,\theta)+\epsilon L(z,\theta), influence function, \mathcal{I}_{up,params}(z)={\frac{d\hat{\theta}_{\epsilon,z}}{d\epsilon}}|_{\epsilon=0}=-H_{\hat{\theta}}^{-1}\nabla_{\theta}L(z,\hat{\theta}), H_{\hat\theta}=\frac{1}{n}\Sigma_{i=1}^{n}\nabla_\theta^{2} L(z_i,\hat\theta) Hessien, \begin{equation} \begin{aligned} \mathcal{I}_{up,loss}(z,z_{test})&=\frac{dL(z_{test},\hat\theta_{\epsilon,z})}{d\epsilon}|_{\epsilon=0} \\&=\nabla_\theta L(z_{test},\hat\theta)^T {\frac{d\hat{\theta}_{\epsilon,z}}{d\epsilon}}|_{\epsilon=0} \\&=\nabla_\theta L(z_{test},\hat\theta)^T\mathcal{I}_{up,params}(z)\\&=-\nabla_\theta L(z_{test},\hat\theta)^T H^{-1}_{\hat\theta}\nabla_\theta L(z,\hat\theta) \end{aligned} \end{equation}, lossNLPer, influence function, logistic regression p(y|x)=\sigma (y \theta^Tx) \sigma sigmoid z_{test} loss z \mathcal{I}_{up,loss}(z,z_{test}) , -y_{test}y \cdot \sigma(-y_{test}\theta^Tx_{test}) \cdot \sigma(-y\theta^Tx) \cdot x^{T}_{test} H^{-1}_{\hat\theta}x, \sigma(-y\theta^Tx) outlieroutlier, x^{T}_{test} x H^{-1}_{\hat\theta} Hessian \mathcal{I}_{up,loss}(z,z_{test}) resistencevariation, \mathcal{I}_{up,loss}(z,z_{test})=-\nabla_\theta L(z_{test},\hat\theta)^T H^{-1}_{\hat\theta}\nabla_\theta L(z,\hat\theta), Hessian H_{\hat\theta} O(np^2+p^3) n p z_i , conjugate gradientstochastic estimationHessian-vector productsHVP H_{\hat\theta} s_{test}=H^{-1}_{\hat\theta}\nabla_\theta L(z_{test},\hat\theta) \mathcal{I}_{up,loss}(z,z_{test})=-s_{test} \cdot \nabla_{\theta}L(z,\hat\theta) , H_{\hat\theta}^{-1}v=argmin_{t}\frac{1}{2}t^TH_{\hat\theta}t-v^Tt, HVPCG O(np) , H^{-1} , (I-H)^i,i=1,2,\dots,n H 1 j , S_j=\frac{I-(I-H)^j}{I-(I-H)}=\frac{I-(I-H)^j}{H}, \lim_{j \to \infty}S_j z_i \nabla_\theta^{2} L(z_i,\hat\theta) H , HVP S_i S_i \cdot \nabla_\theta L(z_{test},\hat\theta) , NMIST H loss , ImageNetInceptionRBF SVM, RBF SVMRBF SVM, InceptionInception, Inception, , Inception591/60059133557%, check \mathcal{I}_{up,loss}(z_i,z_i) z_i , 10% \mathcal{I}_{up,loss}(z_i,z_i) , H_{\hat\theta}=\frac{1}{n}\Sigma_{i=1}^{n}\nabla_\theta^{2} L(z_i,\hat\theta), s_{test}=H^{-1}_{\hat\theta}\nabla_\theta L(z_{test},\hat\theta), \mathcal{I}_{up,loss}(z,z_{test})=-s_{test} \cdot \nabla_{\theta}L(z,\hat\theta), S_i \cdot \nabla_\theta L(z_{test},\hat\theta). Is a dict/json containting the influences calculated of all training data I recommend you to change the following parameters to your liking. Requirements Installation Usage Background and Documentation config Misc parameters we develop a simple, efficient implementation that requires only oracle access to gradients Natural gradient works efficiently in learning. The infinitesimal jackknife. While one grad_z is used to estimate the To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. With the rapid adoption of machine learning systems in sensitive applications, there is an increasing need to make black-box models explainable. On the importance of initialization and momentum in deep learning, A mathematical theory of semantic development in deep neural networks. Or we might just train a flexible architecture on lots of data and find that it has surprising reasoning abilities, as happened with GPT3. A. This will also be done in groups of 2-3 (not necessarily the same groups as for the Colab notebook). Understanding Black-box Predictions via Influence Functions - YouTube AboutPressCopyrightContact usCreatorsAdvertiseDevelopersTermsPrivacyPolicy & SafetyHow YouTube worksTest new features 2022. which can of course be changed. the original paper linked here. How can we explain the predictions of a black-box model? Thus, you can easily find mislabeled images in your dataset, or In Proceedings of the international conference on machine learning (ICML). Often we want to identify an influential group of training samples in a particular test prediction for a given We study the task of hardness amplification which transforms a hard function into a harder one. This code replicates the experiments from the following paper: Understanding Black-box Predictions via Influence Functions. Subsequently, : , , , . Helpful is a list of numbers, which are the IDs of the training data samples Riemannian metrics for neural networks I: Feed-forward networks. to trace a model's prediction through the learning algorithm and back to its training data, A classic result by Radford Neal showed that (using proper scaling) the distribution of functions of random neural nets approaches a Gaussian process. We are given training points z 1;:::;z n, where z i= (x i;y i) 2 XY . A classic result tells us that the influence of upweighting z on the parameters ^ is given by. Linearization is one of our most important tools for understanding nonlinear systems. This alert has been successfully added and will be sent to: You will be notified whenever a record that you have chosen has been cited. The second mode is called calc_all_grad_then_test and Differentiable Games (Lecture by Guodong Zhang) [Slides]. Understanding Black-box Predictions via Influence Functions International Conference on Machine Learning (ICML), 2017. Theano: A Python framework for fast computation of mathematical expressions. In, Mei, S. and Zhu, X. Using machine teaching to identify optimal training-set attacks on machine learners. Koh P, Liang P, 2017. Understanding Black-box Predictions via Influence Functions International Conference on Machine Learning (ICML), 2017. Understanding Black-box Predictions via Influence Functions. 2172: 2017: . We'll also consider self-tuning networks, which try to solve bilevel optimization problems by training a network to locally approximate the best response function. 7 1 . (a) What is the effect of the training loss and H 1 ^ terms in I up,loss? The final report is due April 7. You signed in with another tab or window. Cook, R. D. and Weisberg, S. Characterizations of an empirical influence function for detecting influential cases in regression. That can increase prediction accuracy, reduce affecting everything else. Theano D. Team. On the limited memory BFGS method for large scale optimization. influence function. Understanding black-box predictions via influence functions. Delta-STN: Efficient bilevel optimization of neural networks using structured response Jacobians. >> compress your dataset slightly to the most influential images important for I am grateful to my supervisor Tasnim Azad Abir sir, for his . In Proceedings of the international conference on machine learning (ICML). Thus, in the calc_img_wise mode, we throw away all grad_z Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. Training test 7, Training 1, test 7 . When testing for a single test image, you can then In. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks.See more on this video at https://www.microsoft.com/en-us/research/video/understanding-black-box-predictions-via-influence-functions/ # do someting with influences/harmful/helpful. If you have questions, please contact Pang Wei Koh (pangwei@cs.stanford.edu). One would have expected this success to require overcoming significant obstacles that had been theorized to exist. Either way, if the network architecture is itself optimizing something, then the outer training procedure is wrestling with the issues discussed in this course, whether we like it or not. Metrics give a local notion of distance on a manifold. The first mode is called calc_img_wise, during which the two We'll then consider how the gradient noise in SGD optimization can contribute an implicit regularization effect, Bayesian or non-Bayesian. In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through . In. The next figure shows the same but for a different model, DenseNet-100/12. Visualised, the output can look like this: The test image on the top left is test image for which the influences were multilayer perceptrons), you can use straight-up JAX so that you understand everything that's going on. In this paper, we use influence functions a classic technique from robust statistics to trace a models prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. Biggio, B., Nelson, B., and Laskov, P. Poisoning attacks against support vector machines. James Tu, Yangjun Ruan, and Jonah Philion. initial value of the Hessian during the s_test calculation, this is In, Martens, J. We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. This is the case because grad_z has to be calculated twice, once for To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. This is "Understanding Black-box Predictions via Influence Functions --- Pang Wei Koh, Percy Liang" by TechTalksTV on Vimeo, the home for high quality Pearlmutter, B. While this class draws upon ideas from optimization, it's not an optimization class. We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. Google Scholar Digital Library; Josua Krause, Adam Perer, and Kenney Ng. In this paper, we use influence functions a classic technique from robust statistics to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. Decaf: A deep convolutional activation feature for generic visual recognition. The precision of the output can be adjusted by using more iterations and/or Noisy natural gradient as variational inference. Z. Kolter, and A. Talwalkar. Pang Wei Koh, Percy Liang; Proceedings of the 34th International Conference on Machine Learning, . If the influence function is calculated for multiple above, keeping the grad_zs only makes sense if they can be loaded faster/ Datta, A., Sen, S., and Zick, Y. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. training time, and reduce memory requirements. arXiv preprint arXiv:1703.04730 (2017). $-hm`nrurh%\L(0j/hM4/AO*V8z=./hQ-X=g(0 /f83aIF'Mu2?ju]n|# =7$_--($+{=?bvzBU[.Q. ( , ) Inception, . samples for each test data sample. In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through the learning algorithm and back to its training data, thereby . This is a PyTorch reimplementation of Influence Functions from the ICML2017 best paper: The deep bootstrap framework: Good online learners are good offline generalizers. On the origin of implicit regularization in stochastic gradient descent. In. Influence functions can of course also be used for data other than images, As a result, the practical success of neural nets has outpaced our ability to understand how they work. Google Scholar Krizhevsky A, Sutskever I, Hinton GE, 2012. (b) 7 , 7 . Neural tangent kernel: Convergence and generalization in neural networks. where the theory breaks down, This will naturally lead into next week's topic, which applies similar ideas to a different but related dynamical system. Up to now, we've assumed networks were trained to minimize a single cost function. A tag already exists with the provided branch name. Thus, we can see that different models learn more from different images. C. Maddison, D. Paulin, Y.-W. Teh, B. O'Donoghue, and A. Doucet. PW Koh*, KS Ang*, H Teo*, PS Liang. If the influence function is calculated for multiple Data poisoning attacks on factorization-based collaborative filtering. While these topics had consumed much of the machine learning research community's attention when it came to simpler models, the attitude of the neural nets community was to train first and ask questions later. However, in a lower Data-trained predictive models see widespread use, but for the most part they are used as black boxes which output a prediction or score. ( , , ). We show that even on non-convex and non-differentiable models Deep inside convolutional networks: Visualising image classification models and saliency maps. The power of interpolation: Understanding the effectiveness of SGD in modern over-parameterized learning. 2018. Reconciling modern machine-learning practice and the classical bias-variance tradeoff. fast SSD, lots of free storage space, and want to calculate the influences on He, M. Narayanan, S. Gershman, B. Kim, and F. Doshi-Velez. prediction outcome of the processed test samples. Insights from a noisy quadratic model. Components of inuence. We would like to show you a description here but the site won't allow us. How can we explain the predictions of a black-box model? Understanding Black-box Predictions via Influence Functions Proceedings of the 34th International Conference on Machine Learning . TL;DR: The recommended way is using calc_img_wise unless you have a crazy Despite its simplicity, linear regression provides a surprising amount of insight into neural net training. On robustness properties of convex risk minimization methods for pattern recognition. Ribeiro, M. T., Singh, S., and Guestrin, C. "why should I trust you? We'll see first how Bayesian inference can be implemented explicitly with parameter noise. This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. calculations, which could potentially be 10s of thousands. Neither is it the sort of theory class where we prove theorems for the sake of proving theorems. Existing influence functions tackle this problem by using first-order approximations of the effect of removing a sample from the training set on model . Google Scholar Terry Taewoong Um (terry.t.um@gmail.com) University of Waterloo Department of Electrical & Computer Engineering Terry T. Um UNDERSTANDING BLACK-BOX PRED -ICTION VIA INFLUENCE FUNCTIONS 1 On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks. Gradient-based Hyperparameter Optimization through Reversible Learning. Influence functions help you to debug the results of your deep learning model Class will be held synchronously online every week, including lectures and occasionally tutorials. Strack, B., DeShazo, J. P., Gennings, C., Olmo, J. L., Ventura, S., Cios, K. J., and Clore, J. N. Impact of HbA1c measurement on hospital readmission rates: analysis of 70,000 clinical database patient records. We see how to approximate the second-order updates using conjugate gradient or Kronecker-factored approximations. Jaeckel, L. A. , mislabel . and even creating visually-indistinguishable training-set attacks. Haoping Xu, Zhihuan Yu, and Jingcheng Niu. How can we explain the predictions of a black-box model? In this paper, we use influence functions a classic technique from robust statistics to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. This isn't the sort of applied class that will give you a recipe for achieving state-of-the-art performance on ImageNet. D. Maclaurin, D. Duvenaud, and R. P. Adams. With the rapid adoption of machine learning systems in sensitive applications, there is an increasing need to make black-box models explainable. For toy functions and simple architectures (e.g. Tasha Nagamine, . Biggio, B., Nelson, B., and Laskov, P. Support vector machines under adversarial label noise. We try to understand the effects they have on the dynamics and identify some gotchas in building deep learning systems. The model was ResNet-110. Online delivery. A Dockerfile with these dependencies can be found here: https://hub.docker.com/r/pangwei/tf1.1/. Hopefully this understanding will let us improve the algorithms. In this paper, we use influence functions a classic technique from robust statistics to trace a models prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. Kelvin Wong, Siva Manivasagam, and Amanjit Singh Kainth. Systems often become easier to analyze in the limit. Li, B., Wang, Y., Singh, A., and Vorobeychik, Y. , . However, as stated PVANet: Lightweight Deep Neural Networks for Real-time Object Detection. stream calculate which training images had the largest result on the classification the prediction outcomes of an entire dataset or even >1000 test samples. values s_test and grad_z for each training image are computed on the fly Things get more complicated when there are multiple networks being trained simultaneously to different cost functions. ordered by helpfulness. calculates the grad_z values for all images first and saves them to disk. The most barebones way of getting the code to run is like this: Here, config contains default values for the influence function calculation Check out CSC2541 for the Busy. more recursions when approximating the influence. We motivate second-order optimization of neural nets from several perspectives: minimizing second-order Taylor approximations, preconditioning, invariance, and proximal optimization. To scale up influence functions to modern machine learning settings, This and Hessian-vector products. In this paper, we use influence functions --- a classic technique from robust statistics --- on to the next image. in terms of the dataset. In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. approximations to influence functions can still provide valuable information. Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks. Loss , . S. L. Smith, B. Dherin, D. Barrett, and S. De. Kingma, D. and Ba, J. Adam: A method for stochastic optimization. NIPS, p.1097-1105. /Length 5088 Requirements chainer v3: It uses FunctionHook. , . We use cookies to ensure that we give you the best experience on our website. A. Mokhtari, A. Ozdaglar, and S. Pattathil. For details and examples, look here. Disentangled graph convolutional networks. Most weeks we will be targeting 2 hours of class time, but we have extra time allocated in case presentations run over.

Westdale Properties Net Worth, Marion County Oregon Code Of Ordinances, Does Logan Leave Home And Away, Articles U