Informatics, digital & computational pathology
Digital & computational pathology
Machine learning & deep learning
Convolutional neural networks

Editor-in-Chief: Debra Zynger, M.D.
Jerome Cheng, M.D.

Topic Completed: 17 September 2018

Minor changes: 9 July 2020

Copyright: 2018-2020, PathologyOutlines.com, Inc.

PubMed Search: Convolutional neural networks pathology

Jerome Cheng, M.D.
Page views in 2019: 502
Page views in 2020 to date: 269
Cite this page: Cheng J. Convolutional neural networks. PathologyOutlines.com website. https://www.pathologyoutlines.com/topic/informaticsconvnet.html. Accessed August 11th, 2020.
Definition / general
  • Also referred to as ConvNet
  • Convolutional neural network (CNN) is a machine learning method inspired by the way our visual cortex processes images through receptive fields whereby individual retinal neurons receive stimuli from different regions of the visual field and information from multiple retinal neurons are subsequently passed on to neurons further down the chain (The Data Science Blog: A Quick Introduction to Neural Networks [Acccessed 23 August 2018])
    • Likewise, many CNN architectures have a feed forward neural network architecture composed of convolution and pooling (downsampling) layers, followed by 1 or more fully connected layers (J Cancer 2019;10:4876)
  • Mainly used for image classification, object detection with classification, semantic segmentation and natural language processing
Essential features
  • In machine learning, a convolutional neural network is a class of deep, feed forward artificial neural networks, most commonly applied in pathology to image classification and semantic segmentation (Wikipedia: Convolutional Neural Network [Accessed 27 August 2018])
  • Neural networks, like other supervised machine learning methods, are trained using a dataset with an expected outcome and other parameters that contribute to the prediction of the outcome
    • For instance, a dataset for predicting the presence of hemolysis would have entries for the patient's sex, hemoglobin levels, serum lactate dehydrogenase, serum haptoglobin, indirect bilirubin and the presence or absence of hemolysis as the target feature
    • Variables that contribute to the prediction include the laboratory values and parameter values called weights
    • The value of the weights are adjusted in an iterative manner called backpropagation, where the accuracy of the neural network is assessed through a formula (loss function) and the weights are updated until it arrives at the weight values that give the best prediction accuracy
    • In convolutional neural networks involving images, weights are often in the form of 3 dimensional matrices; the target feature is the class an image belongs to (e.g. benign versus malignant) and the variables that contribute to the prediction are data from the image itself
Terminology
  • Feed forward: refers to how the data flows from 1 layer of the network to a subsequent layer of the network and is further passed on to the next layer of the network after calculations are made in the preceding layer
  • Transfer learning: machine learning models trained to solve 1 type of problem can be reused to solve a problem from a different subject matter, e.g. a machine learning model trained on a nonhistopathological dataset like ImageNet may be utilized to categorize benign and malignant pathology images with some fine tuning or addition of another machine learning layer
  • Semantic segmentation: a pixelwise classification of objects based on image class, e.g. some CNNs may be used to highlight cancerous and stromal regions in an image with different colors (Am J Pathol 2019;189:1686)
Diagrams / tables
  • The following figure illustrates the type of calculations image data goes through in convolution and pooling operations
    • Convolution operations involve an elementwise product between the filter and different segments of equal dimensions from the input matrix
    • Pooling operations perform an aggregate operation (e.g. maximum or average) on a region
    • In the example below, the maximum value was returned from 2 x 2 regions of the input matrix

    Contributed by Jerome Cheng, M.D.

    Convolution and pooling


Images hosted on other servers:

Typical CNN architecture

CNN layers in 3D

Neurons of convolutional layer connected to receptive field

Description of convolutional neural networks
  • Convolution or pooling operations are carried out on information from 1 layer and the results are passed on to a deeper layer of the network
  • Calculations involved in a convolutional neural network (CNN) are complex
    • Fortunately, with all the tools available to us, we do not need to write a program to perform all of these
    • Machine learning frameworks, such as TensorFlow and Pytorch, simplify the process of designing and training CNN models
Image analysis
  • Image analysis through convolutional neural network (CNN) is usually performed on digital slides obtained from a whole slide scanner or an image taken through a CCD device mounted to a microscope
  • Convolutional neural network architecture can be built from scratch or pretrained models can be used for image classification
  • Several pretrained CNN models, such as those based on VGG-16, VGG-19, Inception v3, ResNet50 and DeepLoc, are freely available on the internet and were trained on a particular image subject matter
    • Pretrained VGG-16, VGG-19, ResNet50 and Inception v3 models were trained with the ImageNet image database, comprising over a million images belonging to 1,000 classes of real world objects, such as animals, cars and tables
    • DeepLoc was trained to analyze yeast cell images (Bioinformatics 2017;33:3387)
    • Despite being trained on nonhistological images, they can still be used for analysis of pathology based image datasets
    • VGG-16 and VGG-19 take input images of 224 x 224 pixels
    • Inception v3 takes input images of 299 x 299 pixels
  • Training a new CNN model with a purely histopathological image dataset should further improve prediction accuracy but it would take considerable effort to collect millions of images and model training is expected to take several days to complete
  • Through a process referred to as transfer learning, pretrained CNN models can produce results in minutes; in contrast, training a CNN model from scratch with a large image dataset can take days, even with the aid of a powerful GPU (graphical processing unit)
  • CNNs are widely regarded as black boxes due to the millions of parameters (weights) involved in calculations and difficulty in understanding how these arrive at a prediction; however, in some types of CNNs, class activation maps can highlight which regions in an image contributed most to the prediction (Diagnostics (Basel) 2019;9:38)
  • With the increasing adoption of whole slide digital imaging solutions in pathology departments for research, education and clinical practice, CNN may be used on digital slides to aid in identifying histological structures, such as mitosis, nuclei, cancerous tissue and regions with cancer metastasis in lymph nodes
Applications in pathology
Tools for performing image analysis using convolutional neural networks
  • Python (programming language):
    • Currently the most popular language used for machine learning
    • Has several libraries for convolutional neural network (CNN)
  • Orange (Biolab):
    • Includes an image analysis add-on that can extract features from images using 1 of 7 pretrained CNN models: VGG-16, VGG-19, Inception v3, OpenFace, DeepLoc, Painters, SqueezeNet
    • Extracted features can be combined with machine learning algorithms, such as Random Forest, to create an image classifier (e.g. benign versus malignant lesions)
  • TensorFlow:
    • Open source machine learning library developed by Google
    • Can be used along with Python to develop a CNN architecture for classifying images
  • Keras:
    • High level Python library that works on top of TensorFlow, providing a simpler programming framework for developing deep learning models
  • Pytorch:
    • Open source machine learning framework popular in research
Board review style question #1
Which machine learning method was inspired by the way our visual cortex processes images through receptive fields, whereby retinal neurons receive stimuli from different regions of the visual field and information from multiple retinal neurons are relayed to neurons further down the chain?

  1. Convolutional neural network (CNN)
  2. Logistic regression
  3. Random Forest
  4. Support vector machine
Board review answer #1
A. CNN has a feed forward neural network architecture composed of convolution and pooling (downsampling) layers, followed by 1 or more fully connected layers. Convolution or pooling operations are carried out on information from 1 layer and the results are passed on to a deeper layer of the network. CNN has been used on digital slides to aid in identifying histological structures, such as mitosis, nuclei and regions with cancer metastasis.

Comment Here

Reference: Convolutional neural networks
Board review style question #2
Which layer of a convolutional neural network performs a downsampling operation?

  1. Convolutional layer
  2. Fully connected layer
  3. Pooling layer
  4. ReLU layer
Board review answer #2
C. The pooling layer performs a downsampling operation, e.g. the maximum, minimum or average value of elements belonging to a 2 by 2 matrix may be computed.

Comment Here

Reference: Convolutional neural networks
Back to top