Computer Vision Q&A

Have computer vision or related questions? Ask away and get answers from a team of experienced ML Engineers at Superb AI.

This week’s questions from Ground Truth members and the ML team’s answers! ⬇️ 

❓️Question

Why are variable-size vision networks (e.g. pure convolution + spatial pyramid pooling) not more popular? If you had a dataset with a wide variety of image sizes, what model architecture would you choose? Do vision transformers have an application here?

🤓 Answer

There are a few reasons why variable-size vision networks aren't all that popular:

• Modern deep learning frameworks tend to be optimized for batch processing, and having images in a batch be the same size is essential for efficient GPU memory usage. When variable-sized inputs come into play, things might slow down and become unstable.

• Designing and training variable-size networks can be more challenging than working with fixed-size networks, as they need to handle different scales. This can lead to a more complex and computationally demanding model.

• Transfer learning is incredibly helpful since it allows you to save time, data, and compute resources. However, most pre-trained models have a fixed input size, and adapting them to work with variable sizes without losing the benefits of transfer learning can be difficult.

• With the availability of resizing, padding, and cropping techniques and libraries, it's often much simpler and faster to handle images of different sizes this way rather than developing and training a variable-size network.

So, while variable-size networks might have their place in certain situations, these factors make them less popular choices in the broader deep-learning community.

If you have to work with variable-size images

When you're working with a dataset containing images of various dimensions, and resizing or cropping isn't an option, the choice of model architecture will depend on your specific task, dataset quality, and available computational resources.

Here are some options to consider:

1. RetinaNet or Feature Pyramid Network (FPN): These networks use a pyramid of feature maps with different scales and resolutions, making them great for handling multi-scale input images. They're particularly helpful for object detection tasks with objects of varying sizes in the same image.

2. EfficientNet: these models can adapt to different input sizes and resource constraints while remaining highly effective, thanks to their compound scaling approach. EfficientNet uses a compound coefficient to uniformly scale network width, depth, and resolution in a principled way.

3. U-Net and its variants: U-Net is a popular architecture for segmentation tasks and can handle variable-sized images. The architecture's encoder-decoder structure captures multi-scale contextual information, which can be beneficial for processing images with various dimensions.

4. R-CNN and its variants (Fast R-CNN, Faster R-CNN): These models are primarily designed for object detection tasks and can handle varying input sizes. R-CNNs use region proposals to focus on potentially relevant areas within an image, which can be of different sizes.

What about Vision Transformers?

Vision Transformers show promise in handling various image sizes. By breaking an image into non-overlapping patches and treating them like tokens, ViTs can work with different input sizes with minimal architecture changes. While ViTs can handle this task, keep a few things in mind:

• ViTs typically require larger datasets, so using transfer learning with transformers might be necessary.

• Some ViT implementations use learnable positional embeddings designed for specific input sizes. To handle varying sizes, you might have to modify the positional encoding method.

We also shared a promising new paper in today's newsletter - Vision Transformers with Mixed-Resolution Tokenization. This paper proposes a novel image tokenization scheme, which replaces the standard uniform grid with a mixed-resolution sequence of tokens. Each token represents a patch of arbitrary size.

❓️Question

What DL framework would you use to build reliable systems? Is there a framework that is more robust, compiled, statically typed, or more easily put into production? This question is in contrast to Python which data scientists would be more comfortable with but is somewhat awful to run in production due to dynamic typing, poor standardization, and awkward parallelization patterns.

🤓 Answer

When choosing a robust, production-ready deep learning framework for teams with a mix of deep learning and software engineers, TensorFlow and PyTorch are popular choices. These frameworks offer versatility, extensive documentation, and strong community support, making them attractive options for both research and industry applications.

• TensorFlow is known for its scalability and flexibility, making it a popular choice for large-scale projects. It also has a powerful ecosystem of libraries, like TensorRT for high-performance inference on NVIDIA GPUs. TensorFlow is compatible with multiple programming languages, which makes it easier to integrate with various production systems.

• PyTorch is praised for its dynamic computational graph and ease of use, which makes it a go-to choice for research and experimentation. PyTorch's native support for ONNX allows exporting models to the ONNX format, opening up possibilities for deployment on various platforms. TorchScript, a PyTorch feature, lets you compile and optimize models for production, enhancing the overall performance.

Besides these well-known frameworks, there are other great options designed with production in mind:

• ONNX Runtime ONNX (Open Neural Network Exchange) is a universal format for deep learning models. ONNX Runtime is a performance-oriented inference engine for ONNX models. You can train models with popular frameworks like PyTorch or TensorFlow, then export them to ONNX format for efficient deployment in various languages, including C, C++, Java, and C#.

• Apache MXNet deep learning framework supports multiple languages like Python, C++, and Java. With a focus on performance, scalability, and ease of deployment, MXNet is a great choice for production settings. Additionally, MXNet's hybridization feature optimizes models by compiling them into symbolic graphs, leading to faster execution times.

• TensorRT, created by NVIDIA, stands as a high-performance deep learning inference library tailored for NVIDIA GPUs deployment. It is designed to optimize models that have been trained using TensorFlow or PyTorch. In production settings, TensorRT has the potential to deliver remarkable speed enhancements compared to native TensorFlow or PyTorch implementations.

• TVM, as an open-source deep learning compiler stack, strives to facilitate the efficient deployment of deep learning models across diverse hardware platforms. Capable of accepting models from any framework, such as TensorFlow, PyTorch, or ONNX, TVM compiles these models to run efficiently on a variety of devices, including CPUs, GPUs, and specialized accelerators.

We hope this was helpful!

You can also share your thoughts or follow-up questions in the comment section right here. ⬇️