Steven Braun – Blog

Emacs org-mode: org-emphasize-dwim

Fri, 15 Nov 2024 00:00:00 +0000

I use Emacs for programming, note-taking in org-mode, and scientific writing in LaTeX. Org-mode offers a simple function (org-emphasize &optional CHAR), which inserts an emphasis at a point or region and prompts for CHAR when called interactively. When I write notes or documentation in org-mode, the usual application of org-emphasize is to apply markup such as bold, italic, code, or strikethrough to one or multiple words. Since I’m a previous vim user, I’ve converted to Emacs via the popular Doom Emacs configuration framework, which emphasizes the vim concepts wherever it can. Therefore, my application of org-emphasize to regions usually involves first selecting a region with vim motions. In the case of a single word, this breaks down to ysiw<CHAR>. Citing tpope’s README of surround.vim: It’s easiest to explain with examples. Press ysiw* (you surround inner word) at cursor position [ ]:

Hello [W]orld!

leads to

Hello *[W]orld*!

Since I’m heavily relying on localleader (keybinding ,) for Emacs major-mode functionality, I’m inclined to map essential functions to my localleader group. While I could emulate emacs executing normal mode commands such as ysiw* for bold, ysiw/ for italic, ysiw= for code and ysiw~ for ~~strikethrough~~, I think it is more elegant to introduce a DWIM wrapper to org-emphasize:

(defun sbraun/org-emphasize-dwim (char)
 "DWIM (Do What I Mean) wrapper for org-emphasize.
 If there's an active region, apply emphasis to it.
 Otherwise, apply emphasis to the word at point.
 CHAR is the emphasis character to use."
 (interactive)
 ;; Check if there is an active region (e.g., text is selected).
 (if (use-region-p)
 ;; If a region is active, apply emphasis to the selected region.
 (org-emphasize char)
 ;; Otherwise, apply emphasis to the word at point.
 (save-excursion
 ;; Find the boundaries of the word at point.
 (let ((bounds (bounds-of-thing-at-point 'word)))
 (when bounds
 (goto-char (car bounds))
 (set-mark (cdr bounds))
 ;; Apply emphasis to the selected word.
 (org-emphasize char)
 (deactivate-mark))))))

With this defined, I can add keybindings[1], such as

(map! :localleader
 :map org-mode-map
 (:prefix ("t" "text markup")
 :desc "italic" "i" #'(lambda () (interactive) (sbraun/org-emphasize-dwim ?/))
 :desc "bold" "b" #'(lambda () (interactive) (sbraun/org-emphasize-dwim ?*))
 :desc "code" "c" #'(lambda () (interactive) (sbraun/org-emphasize-dwim ?=))
 :desc "strike" "s" #'(lambda () (interactive) (sbraun/org-emphasize-dwim ?+))))

Back to the original example, pressing ,ti on:

Hello [W]orld!

leads to

Hello /[W]orld/!

which saves me … drum rolls … two keystrokes compared to ysiw* – yay, what a way to procrastinate.

[1] The map! syntax is specific to Doom Emacs, see also: binding-keys.

Tiling macOS: Moving from i3wm to yabai

Thu, 06 Oct 2022 00:00:00 +0000

After using Linux for almost a decade, I’ve finally gotten annoyed at all the little hiccups and issues that arrive from time to time when working in Linux. ArchLinux has taught me more than anything else about the Linux world, its bleeding edge character, and the issues that come along with it. This has brought me to Fedora Linux about two years ago. While more stable in general, even Fedora has its sharp edges here and there. I’ve experienced issues with Bluetooth, audio sinks and sources, printers, and more on a daily to weekly basis. After having less and less available time due to my research, constantly tinkering with my system was no longer an option. Therefore, I’ve decided to finally ditch Linux and give macOS a try.

My Linux workflow was mainly keyboard-driven, using i3wm as a tiling window manager, Emacs as a programming IDE and as a note-taking tool with Org mode and Org Roam, and basically living in the terminal. Therefore, my first goal when switching to macOS was to replicate as much as possible of this exact workflow. While finding that (doom) Emacs worked basically out-of-the-box using the emacs-mac build (brew tap railwaycat/emacsmacport brew install emacs-mac --with-modules) and my Zsh configuration was working without any major changes, finding a workable replacement for i3wm was a much harder task.

Yabai and skhd to the Rescue!

After fiddling around for a few days, I’ve settled with a setup that works really well for me: yabai as a tiling window manager and skhd to define keyboard shortcuts that perform yabai (and some other) commands, replicating most of the functionality that is available in i3wm. In the following, I will go through my yabai and skhd setup and explain how it can replicate the classic i3wm behavior. When giving examples for yabai and skhd commands and configs, these usually go into their respective configuration files at ~/.config/yabai/yabairc and ~/.config/skhd/skhdrc.

Open Terminal

For a terminal-focused workflow, it was important to me to have a quick and simple way to open a new terminal instance bound to my preferred shortcut cmd - return. Kitty allows this via the kitty --single-instance -d ~ arguments. That is, with skhd we can now map cmd - return to this exact call:

cmd - return : kitty --single-instance -d ~

If you prefer iTerm2 over kitty, we can quickly start a new iTerm2 session (as long as there is at least one iTerm2 window already running) with

cmd - return : osascript -e "tell application \"iTerm2\" to set newSession to create window with default profile end tell"

Close Window

To quickly close windows, I map cmd - q to the specific yabai command:

cmd - q : yabai -m window --close

Window Focus

Window management in yabai turns out to be pretty similar to i3wm in practice. Yabai allows compass-like focus commands with yabai -m window --focus <direction>. I typically use the vim-like keys and bind h/j/k/l to west/south/north/east respectively as follows:

cmd - h : yabai -m window --focus west
cmd - j : yabai -m window --focus south
cmd - k : yabai -m window --focus north
cmd - l : yabai -m window --focus east

If you have multiple displays, say next to each other, you can add an alternative command via || if the first command fails. That means if the focus is currently on the east-most window, and we call yabai -m window --focus east, but there is another display right of your current display, the following will handle switching the display as well:

cmd - h : yabai -m window --focus west || yabai -m display --focus west
cmd - l : yabai -m window --focus east || yabai -m display --focus east

Similarly, you can switch stacks conditionally, i.e., first try if you can focus the next or previous window in the current stack and if that fails, conditionally focus the next window south/north:

cmd - j : yabai -m window --focus stack.next || yabai -m window --focus south
cmd - k : yabai -m window --focus stack.prev || yabai -m window --focus north

Move Windows

Similarly, windows can be moved (with my preferred keybinding cmd + shift - h/j/k/l):

cmd + shift - h : yabai -m window --warp west || yabai -m window --display west
cmd + shift - l : yabai -m window --warp east || yabai -m window --display east
cmd + shift - j : yabai -m window --warp south
cmd + shift - k : yabai -m window --warp north

Spaces

The equivalent of i3wm workspaces in macOS are “Desktops”. These can be focused in yabai via the yabai -m space --focus <label> command, where <label> is a tag you assign in your yabairc file:

yabai -m space 1 --label 1
yabai -m space 2 --label 2
yabai -m space 3 --label 3
yabai -m space 4 --label 4
yabai -m space 5 --label 5
yabai -m space 6 --label 6
yabai -m space 7 --label 7
yabai -m space 8 --label 8
yabai -m space 9 --label 9
yabai -m space 10 --label 10

Then in skhdrc, you can use these labels to focus a particular space. Additionally, to simulate the workspace_auto_back_and_forth yes setting of i3wm, we can append the command to focus the most recent space if you press the keybinding for the same space again:

cmd - 1 : yabai -m space --focus 1 || yabai -m space --focus recent
cmd - 2 : yabai -m space --focus 2 || yabai -m space --focus recent
cmd - 3 : yabai -m space --focus 3 || yabai -m space --focus recent
cmd - 4 : yabai -m space --focus 4 || yabai -m space --focus recent
cmd - 5 : yabai -m space --focus 5 || yabai -m space --focus recent
cmd - 6 : yabai -m space --focus 6 || yabai -m space --focus recent
cmd - 7 : yabai -m space --focus 7 || yabai -m space --focus recent
cmd - 8 : yabai -m space --focus 8 || yabai -m space --focus recent
cmd - 9 : yabai -m space --focus 9 || yabai -m space --focus recent

Similar to moving windows around in a specific space, I bind cmd + shift - <label> to moving a window to a particular space:

cmd + shift - 1 : yabai -m window --space 1
cmd + shift - 2 : yabai -m window --space 2
cmd + shift - 3 : yabai -m window --space 3
cmd + shift - 4 : yabai -m window --space 4
cmd + shift - 5 : yabai -m window --space 5
cmd + shift - 6 : yabai -m window --space 6
cmd + shift - 7 : yabai -m window --space 7
cmd + shift - 8 : yabai -m window --space 8
cmd + shift - 9 : yabai -m window --space 9

Spacebar

As an i3bar replacement, there are several options:

simple-bar: A Ubersicht widget, very customizable.
spacebar: A standalone bar application.

To reserve some space in yabai for the bar, you need to configure the height of the external_bar variable in yabai as follows:

yabai -m config external_bar all:0:24

Floating Window Settings

Some windows are just not worth floating and you may collect more of those over time. For this, yabai allows to add rules that disable yabai management for specific apps or windows with titles:

yabai -m rule --add title='Settings$' manage=off
yabai -m rule --add app="^System Preferences$" manage=off
yabai -m rule --add app="^System Information$" manage=off
yabai -m rule --add title="^Preferences$" manage=off
yabai -m rule --add title="^Digital Colour Meter$" manage=off
yabai -m rule --add title="^General.*" manage=off

Additional Resources

My dotfiles are available here, including my old i3wm config, my new yabai config, and the skhd config.

The yabai GitHub repository also hosts a great Wiki that covers everything from installation to configuration.

Optimizing matplotlib visualizations for scientific papers

Wed, 27 Oct 2021 00:00:00 +0000

Without much talk, lets start with a matplotlib example. Let’s say we want to visualize the Focal Loss objective for different values of $\gamma$ w.r.t. the probability of the ground-truth class $p_c^t$ :

If we were to include this figure directly into some LaTeX document, it would look like this:

\documentclass[twocolumn]{article}
\usepackage{blindtext} % Lorem ipsum filler
\usepackage{graphicx} % includegraphics
\begin{document}

\begin{figure}
 \centering
 \includegraphics[width=\linewidth]{./base.png}
 \caption{Lorem ipsum dolor sit amet, consectetuer adipisc-
ing elit. Etiam lobortis facilisis sem. Nullam nec mi et neque
pharetra sollicitudin. Praesent imperdiet mi nec ante. Donec
ullamcorper, felis non sodales commodo, lectus velit ultrices
augue, a dignissim nibh lectus placerat pede.}
\end{figure}

% Some lipsum filler
\Blindtext
\Blindtext
\end{document}

Now, there are a few things that bug me:

The sans-serif font used in the figure stands in contrast to the serif font used in LaTeX documents
The figure font size is smaller than the document font size
The axis grid is missing

Note, that point 2. can be okay if you need to save space and have multiple figures next to each other. If you have enough space, you should always ensure the same font size for all your text (including figures and tables). Furthermore, point 3. can be omitted if the exact data/values are not important. For everything else, axis grids are an easy, non-intrusive hint for the reader for a quick comparisons of values.

The easiest way to accomplish the above is to use the popular SciencePlots python package. It offers multiple matplotlib styles which you can enable via:

plt.style.use(["science", "grid"])

The resulting image, then includes grids, increased font size, and uses a serif font.

To go one step further, one can adjust the legend box frame to look consistent with the axis frame, i.e. use black as color and set the linewidth to 0.5:

legend = plt.legend(fancybox=False, edgecolor="black")
legend.get_frame().set_linewidth(0.5)

Figure Size

It is helpful to adjust the figure size to the actual size available in your LaTeX document. We can find the length of \textwidth by adding the following statement somewhere in the source.

\printinunitsof{in}\prntlen{\textwidth}

This will print the \textwidth variable in inches at the position we have placed it in the document. For the example case with a twocolumn article class, this returns 3.31314 inches. We will now go ahead and make the figure size relative to this base measure by putting the height with a fixed aspect ration in direct relation to the textwidth (and enable an optional scaling factor if necessary for smaller/larger figures):

textwidth = 3.31314
aspect_ratio = 6/8
scale = 1.0
width = textwidth * scale
height = width * aspect_ratio
fig = plt.figure(figsize=(width, height))

PGF Outputs

If we export the figure as .png (or even worse: .jpg) file, the resulting visualization is rasterized and has fixed height and width. This can lead to, depending on the image size, pixelated results when enlarging the figure for an in-detail inspection by the reader. On the other hand, we can simply export the figure either as a .pdf file, or even better, use the .pgf (progressive graphics file) format. The big advance of .pgf over .pdf is the fact that .pgf has no embedded fonts and only tells the latex pgf compiler how to generate the figure from instructions. This leads to the resulting figure in the document body using the very same font for all text as the document text itself.

We can enable the pgf module in matplotlib with the following python preamble:

import matplotlib
matplotlib.use("pgf")
matplotlib.rcParams.update(
 {
 "pgf.texsystem": "pdflatex",
 "font.family": "serif",
 "text.usetex": True,
 "pgf.rcfonts": False,
 }
)

Now we save the figure in the .pgf format instead of the .png format.

- plt.savefig("fig.png")
+ plt.savefig("fig.pgf")

In latex, compiling .pgf files is provided with the pgfplots package.

\usepackage{pgfplots}

And replace the \includegraphics statement with an \input statement as follows

- \includegraphics[width=\linewidth]{fig.png}
+ \input{fig.pgf}

Final Result

With this, we have addressed all issues pointed out earlier on. So let’s compare this directly in the resulting LaTeX output PDF, before (left) and after (right):

Before (left) and after (right) comparison when modifying a matplotlib figure to visually fit into a LaTeX document. We have now fixed the figure dimensions, text size and font, legend borders, and used the pgf format.

The updated figure now looks more polished and visually fits into the context of the LaTeX document with higher consistency. For reference, you can find the python script that generated all above figures here and the LaTeX document here.

A short review of axis-aligned and oriented object detection

Tue, 21 Sep 2021 00:00:00 +0000

This post is going to give a brief introduction to deep models, the history of object detection ranging from classic methods based on hand-crafted features to the latest deep learning object detectors, object detection datasets, and object detection evaluation metrics.

Deep Models Preface

The construction of well performing deep models in complex computer vision tasks is often two-fold. The primary goal is to find a model architecture defined by a directed computation graph $G = \left( V, E \right)$ connecting model inputs $\left\{ \boldsymbol{X}_{0}, \dots, \boldsymbol{X}_{K_{\mathrm{in}}-1} \right\}$ , with $\boldsymbol{X}_{k} \in \mathbb{R}^{d_{0} \times \dots \times d_{D_{k}-1}}$ to nodes $v_{i} \in V$ to model outputs $\left\{ \boldsymbol{Y}_{0}, \dots, \boldsymbol{Y}_{K_{out}-1} \right\}$ with $\boldsymbol{Y}_{k} \in \mathbb{R}^{d_{0} \times \dots \times d_{D_{k}-1}}$ . Each node $v_{i}$ in the graph represents an operation $f$ performed on one or more inputs which in turn generates one or more outputs. The operation can be arbitrary as long as it is differentiable w.r.t. each of its input, i.e.

\frac{\partial f \left( \boldsymbol{X}_{0}, \dots, \boldsymbol{X}_{K_{in}-1} \right) }{\partial \boldsymbol{X}_{i}},\quad 0 \leq i \leq K_{\mathrm{in}}

exists. These operations can be mainly divided into two groups. The first group consists of parametric operations, i.e. the operations' output additionally depends on a set of weights that are adjustable during the optimization step. Prime examples for this group are fully connected layers, which are implemented as affine transformations:

f_{linear} \left( \boldsymbol{x} ; \boldsymbol{W}\right) = \boldsymbol{x} \cdot \boldsymbol{W}, \quad \boldsymbol{x} \in \mathbb{R}^{D_{in}}, \quad \boldsymbol{W} \in \mathbb{R}^{D_{\mathrm{in}} \times D_{\mathrm{out}}} ,

where $\boldsymbol{W}$ is the weights matrix (with an implicit bias encoding) that maps the input from $\mathbb{R}^{D_{\mathrm{in}}}$ to $\mathbb{R}^{D_{\mathrm{out}}}$ , as well as convolution layers, implementing the convolution (which is actually a cross-correlation) operation with weight window, also called kernel map, $\boldsymbol{W} \in \mathbb{R}^{K_{H} \times K_{W}}$ , with $K_{H}$ and $K_{W}$ odd, over an input $\boldsymbol{X} \in \mathbb{R}^{H \times W}$ :

f_{\text{conv2d}, m, n} \left( \boldsymbol{X} ; \boldsymbol{W} \right) = \sum_{i = - \frac{K_{H}-1}{2}}^{\frac{K_{H}-1}{2}} \sum_{j = - \frac{K_{W}-1}{2}}^{\frac{K_{W}-1}{2}} W_{\frac{K_{W}-1}{2} + i, \frac{K_{H} - 1}{2} + j} X_{m-i, n-j} \quad .

Note that this operation in particular is the 2D convolution often used in image processing which is only one of many possible convolution operations. Other convolution operations that are commonly used include 1D and 3D convolution, as well as convolutions with stride, convolutions with dilations, depth-wise, and separable convolutions. Parametric operations have the additional constraint to be differentiable w.r.t. their weights, i.e.

\partial f \left( \boldsymbol{X}_{0}, \dots, \boldsymbol{X}_{K_{\mathrm{in} - 1}} ; \boldsymbol{W} \right) / \partial \boldsymbol{W}

exists.

The second group is formed by non-parametric operations. That is, operations that do not include any learnable weights. Common examples for those are activation functions such as

\begin{aligned} \text{sigmoid} \left( x \right) &= \frac{1}{1 + e^{-x}} \\ \text{tanh} \left( x \right) &= \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} \\ \text{ReLU} \left( x \right) &= \text{max}(0, x) \end{aligned}

which are usually used after affine transformations to achieve non-linearity. Other common non-parametric operations are pooling, normalization (although some normalization techniques such as Batch Normalization [1] do have learnable parameters), dropout [2], and softmax.

The second important step in the development of deep models is the choice of a proper loss function. The loss function $L$ , also called a cost function, measures the overall loss of a model $f$ in taking a decision or action. The goal for the model is to minimize the loss value $L( \boldsymbol{x}, \boldsymbol{y}, f)$ for some input $\boldsymbol{x}$ and the apriori known ground-truth target $\boldsymbol{y}$ . This is achieved by computing the gradient of the loss w.r.t. the weights at each node in the computation graph, using backpropagation and optimizing the weights by taking a descent step in the gradient direction. Simple machine learning tasks use single-term loss functions like the cross-entropy loss for classification or some form of distance metric such as the mean-squared-error for regression. Other tasks may require multiple objectives, such as regressing the coordinates of a bounding box and classifying the object inside that box in object detection. Therefore, loss functions can also be composed of multiple objectives, where each objective $i$ is represented by its loss function $L_{i}$ , weighted by $\lambda_{i} \in \mathbb{R}^{+}$ :

L \left( \boldsymbol{x}, \boldsymbol{y}, f \right) = \sum_{i} \lambda_{i} L_{i} \left(\boldsymbol{x}, \boldsymbol{y}, f \right) .

It is common to include loss terms that are independent of the input and output such as weight decay [3] applying a regularization on the weights, encouraging the model to keep the weights small, as well as gradient penalty [4] which normalizes gradients w.r.t. the inputs, commonly found in successful Generative Adversarial Network [5] architectures.

Quantification of Object Detection Performance

Before we begin to dive into the methodology of object detection, the following will shortly describe common datasets, as well as the de facto standard metric, the mean Average Precision (mAP), in the field of object detection.

Datasets

In computer vision and machine learning in general, the quality of the data which is used to train a model is of utmost importance. The following section lists common datasets for horizontal object detection and oriented object detection.

Horizontal Object Detection

PASCAL VOC

The PASCAL Visual Object Classes (VOC) Challenges¹ [6] (2005 — 2012) includes multiple tasks such as image classification, object detection, semantic segmentation, and action detection. The two most prominent datasets in use for object detection evaluation are VOC07 with 10k training images and 25k annotated objects, and VOC12 with 12k training images and 27k annotated objects. Both datasets contain 20 different classes which are common in everyday life situations such as persons, animals, vehicles, and indoor objects.

ILSVRC

The ImageNet Large Scale Visual Recognition Challenge² [7] (2010 — 2017) is a object detection challenge based on the ImageNet [8] dataset. It contains 200 classes and 517k images with 534k annotated objects beating VOC by two orders of magnitude in scale.

COCO

The Common Objects in Context (COCO)³ [9] (2015 — 2019) is a large-scale object detection, segmentation, and captioning dataset with 80 object categories in 164k images (COCO17) and 897k annotated objects. Before the Open Images Detection challenge (see below), COCO was the most challenging object detection dataset since it contains more object instances per image and more small objects (with a relative image area below 1%), as well as more densely located objects than VOC and ILSVRC.

OID

The Open Images Detection (OID) challenge⁴ [10] (2016 – 2020) released the largest object detection dataset to date in 2018, consisting of 1.9M images with 16M annotations across 600 object categories. Due to the dataset being relatively new, only very few papers publish evaluations for OID.

Oriented Object Detection

The task of oriented object detection requires ground-truth orientation labels for each bounding box. For the above-mentioned datasets, these can only be obtained by applying a minimum-bounding-rectangle algorithm on the complex hull of the segmentation map of each object. Alternatively, oriented object detection datasets have been gathered, as listed below.

DOTA

The Dataset for Object Detection in Aerial Images (DOTA) [11] was released as part of the Object Detection in Aerial Images (ODAI)⁵ challenge in early 2018. In the first version (1.0) of the dataset, a total of 2806 images are collected from Google Earth and vary in size between $800 \times 800$ and $4000 \times 4000$ pixels, annotated by 188k objects in total. Each object instance annotation consists of an arbitrary quadrilateral, i.e. 8 degrees of freedom (four pairs of $x$ - and $y$ -coordinates), as well as one label from a set of 15 possible object categories. While all publications on oriented object detection evaluate on version 1.0 of DOTA, the dataset authors have additionally published version 1.5 which introduces an additional class and increases the number of annotations on the existing image base to 403k.

HRSC2016

The High Resolution Ship Collection (HRSC)⁶ [12] dataset was collected from Google Earth and consists of 1.7k satellite images. Each image can contain multiple ships and each ship is annotated with a 5-tuple describing the pixel location, width, height, and rotation angle. Additionally, each ship is annotated with a label for the ship class, a specific category, and a ship type.

ICDAR

The International Conference on Document Analysis and Recognition (ICDAR)⁷ offers the ICDAR2015 [13] challenge on incidental scene text detection containing 1.7k everyday scene images. Each text instance is annotated with a quadrilateral (8 degrees of freedom) specifying the arbitrary bounding box and the actual text content.

FDDB

The Face Detection Data Set and Benchmark (FDDB)⁸ [14] is a dataset of faces, designed to study the problem of unconstrained face detection. The annotations consist of 5.2k faces in 2.8k images, where each instance is described by a 5-tuple of rotated ellipsis.

Measuring Detection Accuracy

The Intersection over Union metric can be computed by measuring the proportion of the intersection between two bounding boxes w.r.t. their joint area, see (a). The IoU value ranges between $0.0$ when $A \cap B = \emptyset$ (b) and $1.0$ when $A \cap B = A \cup B$ , implying that $A = B$ (c).

The most common evaluation metric in object detection is the mean Average Precision (mAP), originally introduced in VOC07. The mAP score is computed as the average object detection precision, i.e. “What proportion of positive detections was actually successful?”, over different recall values, i.e. “What proportion of actual positives was detected successfully?”, evaluated for each object class separately and averaged afterward. The values for precision and recall can be computed as follows:

\begin{aligned} \text{Precision} &= \frac{\text{TP}}{\text{TP + FP}} \\ \text{Recall} &= \frac{\text{TP}}{\text{TP + FN}} , \end{aligned}

where TP is the number of true positives, while FP and FN are the numbers of false positives and false negatives respectively. The natural follow-up question is how a detection match and miss is decided. In a binary classification problem, we simply check for equality between the predicted and the target label. In the setting of object detection, the targets and predictions consist of tuples of coordinates that define a bounding box. Hence it is necessary to define a rule at which point the prediction matches the target box in the two-dimensional image space. This rule can be expressed as a hard threshold for the so-called Intersection over Union (IoU) value which measures the relative overlap between the two boxes $|A \cap B|$ w.r.t. their common covered area $|A \cup B|$ (see Figure above):

\text{IoU}\left( A, B \right) = \frac{|A \cap B|}{|A \cup B|} .

The IoU score ranges between a value of 0.0 when there is no overlap between the two boxes ( $A \cap B = \emptyset$ , Figure (b)) and 1.0 when the boxes are equal ( $A \cap B = A \cup B$ , Figure (c)). For a fixed IoU score threshold $\tau$ , we can now count the necessary statistics to compute the precision and recall values as follows:

True Positives: Number of predicted boxes $A$ that fulfil $\text{IoU}\left( A, B \right) \geq \tau$ for at least a single target box $B$ , i.e. those objects which are correctly localized.
False Positives: Number of predicted boxes $A$ that fulfil $\text{IoU}\left( A, B \right) < \tau$ for all target boxes $B$ , i.e. those predictions that did not sufficiently overlap with any target.
False Negatives: Number of target boxes $B$ that fulfil $\text{IoU}\left( A, B \right) < \tau$ for all predicted boxes $A$ , i.e. those targets that had no sufficient overlap with any prediction.

An example precision-recall curve. The blue line represents the true PR-Curve, while the dotted orange line is the 11-point interpolation which at recall point $\tilde{r}$ uses the maximum precision value $p \left( r \right)$ for all $r \geq \tilde{r}$ with $\tilde{r} \in \left\{ 0.0, 0.1, \dots, 1.0\right\}$ . The Average Precision score is equal to the area under the 11-point interpolation of the precision-recall curve.

Following a sorting procedure of each prediction based on object classification confidence value, we can then generate the precision-recall curve by counting the number of TP, FP and FN cumulatively along the confidence-ascending list of predictions as shown with the blue line in the above figure. The Average Precision (AP) score reflects the area under the precision-recall curve. To be more robust against small changes in prediction confidences and the following change in the precision-recall curve, the area under the curve is interpolated using an 11-point average, i.e. for each recall value $\tilde{r}$ in the range $\left[ 0, 1 \right]$ with a step size of 0.1, the according precision $p \left( \tilde{r} \right)$ is set to be the maximum precision over all $r \geq \tilde{r}$ :

\text{AP} = \int_{0}^{1} p \left( r \right) dr \approx \frac{1}{11} \sum_{\tilde{r} \in \{0.0, 0.1, \dots, 1.0\}} \max_{r \geq \tilde{r}} p \left( r \right)

The extension of the Average Precision to a multi-class problem is called the mean Average Precision (mAP):

\text{mAP} = \frac{1}{|C|} \sum_{c \in C} \text{AP}_{c} ,

where $C$ is the set of available object classes and $\text{AP}_{c}$ is the Average Precision score for a specific class $c$ . The IoU based mAP score has become the prime metric for object detection evaluation. Nevertheless, it is common to report a batch of scores for different IoU thresholds, namely $\text{mAP}_{0.5}$ for a 0.5-IoU threshold, $\text{mAP}_{0.75}$ for a 0.75-IoU threshold, and $\text{mAP}_{\left[0.5, 0.95\right]}$ , sometimes abbreviated as mAP or simply AP when clear from context, for an averaged mAP score over 10 equally distanced IoU thresholds between 0.5 and 0.95. Another distinction is the separation of $\text{mAP}$ scores into different object sizes as is common in the COCO benchmark: $\text{mAP}^{small}$ , $\text{mAP}^{medium}$ , and $\text{mAP}^{large}$ are mAP values for objects with an $\text{area} < 32^{2}$ , $32^{2} < \text{area} < 96^{2}$ , and $96^{2} < \text{area}$ , in pixel^2^ respectively.

A History of Object Detection

As modern deep learning-based object detection borrows many techniques from traditional approaches, it is important to quickly summarize these before moving on to more recent ones. In an era before the prominent rise of deep learning models in the last decade, robust image representation optimized towards a specific task could not be simply learned from the data but had to be handcrafted and designed sophisticatedly. As used in [15, 16, 17] and later successfully optimized in the first real-time application for human face detection by [18], the basics of object detection used to be a straight forward approach: A sliding window is used to detect object instances in all locations and scales of an image. Each image sub-region under the current window position is then used to compute so-called Haar-like features (similar to Haar wavelets) and classifiers are then learned to distinguish between positive samples, i.e. feature representations of sub-regions which contain the object, and negative samples, those that count towards the background and are not of interest.

History of Oriented Gradients feature transformation applied on a test image. Left: Test image. Middle: HOG descriptors weighted by positive SVM weights. Right: HOG descriptors weighted by negative SVM weights. Source: [19].

Introduced in [19], Histogram of Oriented Gradients (HOG) have been developed as an important improvement over scale-invariant feature transform (SIFT) [20] descriptors. The main idea behind HOG features is that local object shapes and appearances in an image can be expressed as a distribution of color intensity gradients. The above figure shows a test image of a pedestrian in and the HOG descriptors weighted by positive and negative Support Vector Machine (SVM) weights, which were used to classify the presence of a pedestrian, in and respectively. HOG features became an important foundation of many object detectors [21]; [22]; [23], as well as other computer vision tasks.

The peak of traditional object detection methods was reached with the Deformable Part-based Model (DPM) proposed by , winning the VOC-07, -08, and -09 detection challenges. It is built on the foundations of the HOG detector and views training as the task to learn how to decompose an object while inference is an ensemble of detections of different object parts. Detecting a person would then be translated into the decomposed detection of a head, legs, hands, arms, and body, which was also called the “star-model” in [21]. [24] later improved this to “mixture models” [22]; [24]; [25], coping with objects of larger variation.

Deep Learning based Object Detection

After the success of DPMs, improvements in object detectors stagnated. With the comeback of convolutional neural networks (CNN) [26] in 2012, deep architectures have been developed to learn robust and high-level task agnostic feature representations of images that easily superseded hand-crafted ones. Unsurprisingly, the field of object detection has quickly gained new traction due to successful deep models. This section goes into more detail on different approaches which can be grouped into two-stage detectors and one-stage detectors. The former can be split into two steps that divide the candidate region generation and the actual location regression and object classification while the latter implements an end-to-end solution using a single deep neural network.

Two-Stage Detectors

Two-stage object detectors follow a detection paradigm of a separated (1) proposal detection step, where likely object locations are determined and (2) a verification step, where each proposal is classified into one of the possible classes of objects and additionally the proposed location is fine-tuned.

Regions with CNN Features: R-CNN

R-CNN object detection system overview. The system (1) takes an input image, (2) extracts region proposals, (3) warps and forwards each proposal through a pre-trained CNN to obtain feature representations and finally (4) classifies each output using class-specific SVMs. Source: [27].

The first of its kind in the two-stage category of object detectors was R-CNN (Regions with CNN features) by . R-CNN starts with the generation of object proposals that serve as candidates for processing. These are obtained using the selective search algorithm [28] which is a region proposal procedure that computes hierarchical groupings of similar regions based on size, shape, color, and texture. Each object proposal is then warped into a fixed predetermined image size and forwarded through a CNN model, which is pre-trained on ImageNet, to extract a fixed 4096-dimensional feature vector. Afterward, class-specific SVMs perform the object recognition task by scoring each region proposal with their respective class. Finally, a greedy non-maximum suppression is applied, rejecting regions of high IoU values with other regions of the same class achieving higher SVM scores than a learned threshold. The above figure gives a sketch of the inference pipeline. Additionally, the CNN can be fine-tuned on other datasets. To improve the object localization, [27] have applied a separate bounding box regression stage (a similar strategy was introduced in [29]) that uses class-based regressors to predict new bounding boxes based on the CNN features. R-CNN broke the stagnation in the field of object detection by pushing the VOC07 $\text{mAP}_{0.5}$ score from 33.7% of DPM-v5 [30] to 58.5%.

Fast R-CNN

The architecture of Fast R-CNN. The full input image is fed through a CNN to generate feature maps. Based on RoIs, each feature map is pooled into a RoI feature vector and fed into a sequence of fully-connected layers with a classification (object classes) and regression (object localization) head. Source: [31].

R-CNN was superseded by Fast R-CNN [31]. In Fast R-CNN, the full input images are forwarded through a CNN backbone to produce features maps. For each object proposal output of the selective search algorithm, a region of interest (RoI) pooling layer extracts a fixed-size feature vector from the feature map, inspired by spatial pyramid pooling in SPPNet introduced by [32]. Then, each vector is fed into a sequence of fully-connected layers which then split into two heads, one for softmax classification outputting a probability vector $\boldsymbol{p}$ of length $|C|+1$ for the possible classes (with an additional class for the background), and one for the bounding box regression outputting a bounding box offset $\boldsymbol{t}^{c} = ( t_{x}^{c}, t_{y}^{c}, t_{w}^{c}, t_{h}^{c}, )$ for each of the $|C|$ object classes (see figure above). To jointly train for classification and bounding box regression, a multi-task loss $L$ on each labeled RoI is introduced:

L\left(\boldsymbol{p}, u, \boldsymbol{t}^u , \boldsymbol{v}\right) = L_{cls}(\boldsymbol{p}, u) + \lambda\left[u \geq 1\right]L_{loc}\left(\boldsymbol{t}^u, \boldsymbol{v}\right) ,

where $u$ is the ground-truth label, $\boldsymbol{v}$ is the ground-truth bounding box regression target, the Iverson bracket indicator function $[u \geq 1]$ evaluates to 1 when $u \geq 1$ and 0 otherwise, and the hyperparameter $\lambda$ controls the balance between the two task losses. For classification the binary cross-entropy loss and for bounding box regression, the SmoothL1 loss is used to accumulate over the coordinates. For background RoIs ( $u = 0$ ) $L_{loc}$ is ignored due to missing ground-truth references.

This approach is closer to an end-to-end solution than its predecessors as it gets rid of the multi-stage pipeline and can be trained given only the input image and the object proposals coming from an off-the-shelf algorithm. Fast R-CNN improves training time by a factor of 9 (with a VGG16 backbone) and testing time by a factor of 213, and pushes the $\text{mAP}_{0.5}$ score on VOC07 to 70.0%.

Faster R-CNN

The End-to-end unified architecture of Faster R-CNN including an ‘attention’ module called Region Proposal Network (RPN) generating likely object proposals (location and objectness) based on $k$ predefined anchor boxes. Source: [33].

Fast R-CNN was expanded by Faster R-CNN [33], replacing the external object proposal stage with an end-to-end approach (see figure above). [33] have introduced a so-called Region Proposal Network (RPN) serving as an “attention” model which is a fully convolutional network that takes an image of any size as input and predicts a set of bounding box proposals. Each proposal is attached with an objectness score that measures the membership to a set of object classes against the background class. Instead of generating final bounding box locations of the object proposals, [33] introduce the notion of anchors, making Faster R-CNN the first anchor-based detector. These are used as a baseline reference box for which an offset has to be regressed (see figure above). Anchor-boxes are predefined with different scales (e.g. $32\times32$ , $64\times64$ , …) and aspect ratios (e.g. 1:1, 1:2, 2:1, 5:1, …). For each position in a convolutional feature map in the RPN, a sliding window approach computes a vector of $2k$ objectness scores, one for the class “object” and one for the class “background”, as well as $4k$ bounding box offsets where $k$ is the number of different anchors (the product of the number of anchor scales and the number of anchor aspect ratios). Therefore, the RPN generates $W \cdot H \cdot k$ proposals, where the intermediate feature map is of size $W \times H$ .

Embedding the region proposal generation into the network stack has improved the VOC07 $\text{mAP}_{0.5}$ to 73.2% (COCO $\text{mAP}_{0.5}=42.7\%$ , COCO $\text{mAP}_{\left[0.5, 0.95\right]}=21.9\%$ ). Faster R-CNNs therefore became the first end-to-end and the first near-realtime object detector (17fps with a ZFNet [34] backbone). Computational redundancies at the subsequent detection stage have later been reduced in RFCN [35] using fully convolutional networks, and Light-Head R-CNN [36] thinning out the prediction heads and replacing the CNN backbone with smaller networks (e.g. Xception [37]).

Faster R-CNN has been extended in Feature Pyramid Networks (FPN) [38]. Feature pyramids are a principal component in computer vision tasks for objectives that have to be solved at multiple scales. Until the development of FPN, deep learning-based object detectors have been avoiding feature pyramids, mostly due to their computational complexity and high memory usage. FPNs utilize the inherent multi-scale hierarchy present in deep convolutional networks and introduce feature pyramids with almost no extra cost. Instead of only using the very last output in the convolutional layer sequence, FPNs introduced a top-down architecture with lateral connections (see figure above) which extracts intermediate feature maps from another deep network (in this context called backbone). This architecture allows building high-level semantics at all scales, significantly improving scores on COCO to $\text{mAP}_{0.5}=59.1\%$ and $\text{mAP}_{\left[0.5, 0.95\right]}=36.2\%$ . Feature Pyramid Networks have since become one of the basic building blocks for many newer object detectors.

One-Stage Detectors

In contrast to two-stage object detectors, a parallel line of development took place with a very different approach. Instead of having a separate stage in the network that proposes where an object is likely, one-stage detectors predict object locations and classes on a grid that can be mapped onto the input image in a single pass which is methodologically simpler and computationally faster. This also allows the object detector to be trained in a simple end-to-end fashion, therefore optimizing the whole network in a unified way towards the defined objective, unlike two-stage methods which usually have to define freezing phases for different network parts with different loss functions to achieve a stable training (see “4-Step Alternating Training” in [33].

You Only Look Once (YOLO)

YOLO divides the input image into a grid of $S\times S$ patches. Each grid cell then predicts $B$ bounding boxes, confidences for the boxes, and $C$ class probabilities. Source: [39].

The most prominent and also first one-stage object detector was YOLOv1, proposed by . The network separates the input image into a grid of $S \times S$ patches and predicts bounding boxes, object confidences, and class probabilities for each patch at the same time (see figure above), reaching $\text{mAP}_{0.5} = 63.4\%$ on VOC07. YOLOv1 was improved in YOLOv2 [40], adapting anchor boxes from Faster R-CNN, achieving a $\text{mAP}_{0.5}$ score of 78.6% on VOC07, and $\text{mAP}_{0.5}=44.0\%$ and $\text{mAP}_{[0.5,0.95]}=21.6\%$ on COCO. It nevertheless suffered from weak localization accuracy for small objects, which was addressed in YOLOv3 [41] by making use of features from multiple scales, similar in concept to feature pyramids in FPNs ( $\text{mAP}_{0.5} = 57.9\%$ and $\text{mAP}_{[0.5,0.95]} = 33.0\%$ on COCO).

Single Shot Detection (SSD)

An architectural comparison of SSD (top) and YOLOv1 (bottom). SSD works without any fully-connected layers and adds convolutional feature layers on-top of the backbone network to predict offsets for a set of anchor boxes at multiple feature scales. Evaluated mAP values are for the VOC07 dataset. Source: [42].

SSD, proposed by , was the second one-stage object detector after YOLOv1. Its major contribution is using default boxes of different scales on different intermediate feature maps instead of only using the last layer (see figure above). Taking multiple feature scales into account, SSD offers advantages in terms of detection speed and accuracy over YOLOv1, especially for small objects (VOC07 $\text{mAP}_{0.5}=76.8\%$ , COCO $\text{mAP}_{0.5}=46.5\%$ , COCO $\text{mAP}_{\left[0.5, 0.95\right]}=26.8\%$ , and a fast version running with 59FPS VOC07 $\text{mAP}_{0.5}=74.3\%$ depicted in the above figure).

RetinaNet

In RetinaNet [43], [43] discover that one-stage detectors have trailed two-stage detectors in terms of accuracy due to their unmanaged class imbalance of positive and negative samples. Two-stage detectors can use sampling heuristics such as a fixed foreground-to-background ratio (typically 1:3), or online hard example mining (OHEM) [44] to maintain the class balance between foreground (positive samples) and background (negative samples). Since one-stage detectors perform a single pass and have no stage to filter possible object proposals, they evaluate about $10^4$ to $10^{5}$ candidate locations per image while only a few locations contain objects, causing the following two issues: (1) inefficient training since most locations are easy to pick backgrounds and do not contribute any useful learning signal and (2) negative sample numbers overweight positives ones by a large margin, leading to degenerate models as the training can only adjust to few positives in comparison to negatives. To tackle this issue in one-stage detectors, the authors propose the so-called Focal Loss which represents a modified version of the cross-entropy loss. The main idea of Focal Loss is to automatically down-weight the contribution of easy examples during training and focus the model on hard examples with low confidences. Using feature pyramids from FPN [38] (with a ResNeXt-101-FPN [45] backbone) and anchor boxes from Faster R-CNN, RetinaNet managed to beat previous state-of-the-art results with a COCO mAP of 40.8%, surpassing the best one-stage detector DSSD513 [46] (with a ResNet-101-DSSD backbone) at 33.2% mAP as well as the best two-stage detector Faster R-CNN (with an Inception-Resnet-v2-TDM [47] backbone) at 36.8% mAP.

Anchor-Free Object Detection

Although anchor-based methods have shown great success, they come with increased methodological and computational complexity. [33] show in [33] that detection performance is sensitive to anchor box size, aspect ratio and number. Therefore, hyper-parameters need to be additionally tuned to find anchors appropriate for the specific dataset. Moreover, anchor-based detectors have difficulties with objects of large shape variations, as well as small objects. The need for fine-tuning moreover impedes the model’s generalization abilities, as anchors need to be redesigned for new tasks. Most anchor boxes are labeled as negative samples during training since anchors are required to be placed densely on the input image (FPN [38] e.g. places 180k anchors) leading to only a few anchors overlapping with positive samples. Logically, the question of whether anchor-based solutions are the optimal way to solve object detection arises. The recent field of anchor-free object detection tries to find answers to this exact question. First results show, that approaches without object anchors are competitive and capable of beating state-of-the-art anchor-based models, with the advantage of being faster and methodologically less complex.

Early approaches such as DenseBox [48], the first unified end-to-end fully convolutional detector, UnitBox [49] tackling the localization with an IoU-based loss, YOLOv1 [39] focusing on real-time object detection, were made but without success in surpassing anchor-based systems such as Faster R-CNN at that time.

CornerNet [50] approaches the bounding box prediction by predicting a pair of keypoints, the top-left, and bottom-right corners. The network predicts a heatmap for the top-left corner, a heatmap for the bottom-right corner, and an embedding for each corner which is supposed to group pairs of corners that belong to the same bounding box, minimizing embedding vector distance between pairs. Embeddings are produced using the Associative Embedding [51] technique to separate different instances. CornerNet achieves a mAP of 42.2% on COCO. CenterNet [52] builds on top of CornerNet and includes the prediction of a center keypoint used to perform center pooling. Additionally, CenterNet also introduces cascade corner pooling as an extension to corner pooling from CornerNet which performs max-pooling first along the bounding box borders and then along the orthogonal row/column towards the region center. This addresses the problem that CornerNet and most other one-stage detectors lack an additional look into the cropped regions by exploring the visual patterns within each predicted bounding box. CenterNet achieves a mAP of 47.0% on COCO, giving a significant improvement of 4.8% mAP over CornerNet.

ExtremeNet [53] on the other hand is motivated by who proposed to annotate bounding boxes by marking the objects’ four extreme points: top, bottom, left, right. In ExtremeNet, object extreme keypoints are predicted as follows. First, four multi-peak heatmaps, one for each extreme, are predicted for each object category to generate possible extreme points. Then, each combination between a top, bottom, left, and right extreme point is being generated and their geometric center is being compared to a fifth heatmap that generates center keypoints. If the geometric center is close to one of the peaks in the center heatmap (distance above a fixed threshold), the extreme point combination is valid and an object is predicted.

FCOS [54] uses a fully convolutional one-stage detection approach introducing the notion of center-ness, which depicts the normalized distance from the location to the center of the object that the location is responsible for. The center-ness is used to adjust the classification confidence and thus helps to suppress low-quality detected bounding boxes and improves overall performance by a large margin ( $\text{mAP}_{[0.5,0.95]}$ on the COCO minival validation subset of 37.1% with, compared to 33.5% without the center-ness branch).

FoveaBox [55] is a single unified network, composed of a backbone network and two task-specific sub-networks, following the RetinaNet [43] design. Its main contribution is the assignment of different object scales to different feature pyramid outputs, i.e. each pyramid layer is responsible for a certain interval of object sizes.

Oriented Object Detection

Comparison of axis-aligned (top) and oriented (bottom) bounding boxes in object detection on aerial view imagery. It becomes clear that the oriented bounding box representation is superior for oriented objects and generates tighter boxes that better cover the true area of the object.

Oriented object detection has recently gained more attention in computer vision for aerial imagery, scene text, and face detection. In oriented object detection, bounding boxes are not bound to be strictly horizontal and vertical along the x- and y-axis but can be either rotated by an arbitrary angle or defined by an arbitrary quadrilateral consisting of the four corners. This allows for tighter bounding boxes, especially in densely populated regions as well as for objects that are not parallel to the physical x-y plane, where x is the horizontal and y is the vertical axis. The above figure shows the effect of using oriented bounding boxes, compared to horizontal bounding boxes. The top row shows axis-aligned bounding boxes on ships, vehicles, and airplanes in aerial view imagery while the bottom row shows their respective oriented bounding boxes. It is clear that the oriented bounding box representation is superior for oriented objects and generates tighter boxes that better cover the true area of the object.

Two-Stage Detectors

Since this field has only recently received more attention, current approaches are usually based on methods that are successful in horizontal object detection. A straightforward approach to tackle oriented object detection is to extend the prediction of a bounding box with an additional parameter $\theta$ , determining the rotation angle. did so by using a similar network as Faster R-CNN, expanding the bounding box priors by multi-angle anchors including rotations between 0 $^{\circ}$ and 180 $^{\circ}$ , thus increasing the anchor hyperparameter set significantly. Similarly, propose Rotation Region Proposal Networks (RRPN), adapting the Region Proposal Network from Faster R-CNN, designed to generate RoIs with angle rotation information.

Inspired by Mask R-CNN [56], proposed a semantic segmentation-guided RPN (sRPN) using the atrous spatial pyramid pooling (ASPP) module from [57] to suppress background clutter and a RoI module that fuses multi-level outputs from an FPN. Similarly, SCRDet++ [58] introduced the idea of instance-level denoising on the feature maps into object detection to enhance the detection of small and cluttered objects, common in satellite images.

RoI Transformer [59] is another two-stage anchor-based detector where the geometric transformation from horizontal bounding boxes to oriented bounding boxes is learned and applied to the RoI output of an RPN, such as in Faster R-CNN.

argue that a five-point representation of oriented bounding boxes can cause training instability as well as performance decreases. They ascribe this to the loss discontinuity which results from the natural periodicity of angles and therefore possible exchanges of width and height in the box representation. To circumvent this discontinuity, [60] propose to use the quadrilateral representation in combination with a loss modulation that greedily minimizes the loss from the set of possible edge assignments.

One-Stage Detectors

Salience Biased Loss [61] model architecture. The lower network part (green) is the RetinaNet backbone generating multiscale feature maps while the upper part (blue) is a second network which estimates the salience maps used to adapt the focal loss to the difficulty of the current image. Source: [61].

Similar to [43], [61] propose a novel loss function based on salience information directly extracted from the input image. Like the Focal Loss, the proposed Salience Biased Loss (SBL) treats training samples differently according to the complexity (saliency) of an image. This is estimated by an additional deep model trained on ImageNet [8] in which the number of active neurons across different convolution layers are measured (see figure above):

S = \frac{1}{C \cdot W \cdot H} \sum_{c=1}^{C} \sum_{w=1}^{W} \sum_{h=1}^{H} f\left( \boldsymbol{x} \right)_{c,w,h} ,

where $S$ is the average activation value across the layer and $f$ is a convolution operation with an output feature map of size $C \times W \times H$ . The idea is that with increasing complexity, more neurons will be active. The saliency then scales an arbitrary base loss function to adapt the importance of training samples accordingly.

Architecture illustration of S2A-Net consisting of a backbone pretrained on ImageNet, the Feature Pyramid Network to extract multiscale features, the Feature Alignment Module to generate oriented anchors using aligned convolutions and the Oriented Detection Module using Active Rotating Filters. Source: [62].

Han et al. [62] approach to solve the discrepancy of classifications score and localization accuracy and ascribe this issue to the misalignment between anchor boxes and the axis-aligned convolutional features. Hence they propose two modules (see the figure above for the full architecture): The Feature Alignment Module (FAM) generates high quality oriented anchors using their Anchor Refinement Network (ARN) and adaptively aligns the convolutional features according to the generated anchors using an Alignment Convolution Layer (ACL).

Sampling locations of different convolution operations using a $3 \times 3$ kernel. Figure (a) is the default convolution while (b) portrays deformable convolutions [63], enabling learnable sampling offsets for each of the 9 locations. Figure (c) and (d) are aligned convolutions (AlignConv) which in turn are deformable convolutions but restricted to a global translation, rotation and scaling w.r.t. the full sampling window and are supposed to align the convolution operation to oriented anchor boxes. Source: [62].

The ACL is a restricted Deformable Convolution Layer [63] in the sense that it learns the same translation, scaling, and rotation for each sampling location (see figure above). The second proposed module is the so-called Oriented Detection Module (ODM). Introduced in Oriented Response Networks (ORN) [64], [62] make use of Active Rotating Filters (ARF) which is a $k \times k \times N$ convolutional filter that rotates the features $N - 1$ times, generating an output feature map of $N$ orientation channels, thereby encoding $N$ orientations directly into the feature maps.

In [65], the authors tackle the issue of discontinuous boundary effects on the loss due to the inherent angular periodicity and corner ordering by transforming the angular prediction task from a regression problem into a classification problem. They devise the Circular Smooth Label (CSL) technique which handles the periodicity of angles and raises the error lenience to adjacent angles.

One-Stage Anchor-Free Detectors

The following contributions go one step further and remove the concept of anchors, generating predictions on a dense grid over the input image.

IENet [66] is based on the one-stage anchor-free fully convolutional detector FCOS. The regression head from FCOS is extended in IENet by another branch that regresses the bounding box orientation, using a self-attention mechanism that incorporates the branch feature maps of the object classification and box regression branches.

Axis-Learning [67] also builds on the dense sampling approach of FCOS and explore the prediction of an object axis, defined by its head point and tail point of the object along its elongated side (which can lead to ambiguity for near-square objects). The axis is extended by a width prediction which is interpreted to be orthogonal to the object axis.

In PIoU [68] the authors argue, that a distance-based regression loss such as SmoothL1 only loosely correlates to the actual IoU measurement, especially in the case of large aspect ratios. Therefore, they propose a novel Pixels-IoU (PIoU) loss, which exploits the IoU for optimization by pixelwise sampling, improving detection performance on objects with large aspect ratios dramatically.

P-RSDet [69] replaces the Cartesian coordinate representation of bounding boxes with polar coordinates. Therefore, the bounding box regression happens by predicting the object’s center point, a polar radius, and two polar angles. Furthermore, to express the geometric constraint relationship between the polar radius and the polar angles, a novel Polar Ring Area Loss is proposed.

An alternative formulation of the bounding box representation is defined in O2-DNet [70]. Here, oriented objects are detected by predicting a pair of middle lines inside each target, showing similarity to the extreme keypoint detection schema proposed in ExtremeNet.

References

Sergey Ioffe, Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Proceedings of the 32nd International Conference on Machine Learning, 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 2014.
Anders Krogh, John A Hertz. A Simple Weight Decay Can Improve Generalization. Proceedings of the 4th International Conference on Neural Information Processing Systems, 1991.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, Aaron C Courville. Improved Training of Wasserstein GANs. Advances in Neural Information Processing Systems, 2017.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, et al. Generative Adversarial Nets. Neural Information Processing Systems (NeurIPS), 2014.
Mark Everingham, Luc Gool, Christopher K Williams, John Winn, Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. Int. J. Comput. Vision, 2010.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, et al. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
J Deng, W Dong, R Socher, L.-J. Li, K Li, L Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. Proc. of CVPR, 2009.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, et al. Microsoft COCO: Common Objects in Context. Computer Vision – ECCV 2014, 2014.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, et al. The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, et al. DOTA: A Large-Scale Dataset for Object Detection in Aerial Images. Proc. of CVPR, 2018.
Zikun Liu, Liu Yuan, Lubin Weng, Yiping Yang. A High Resolution Optical Satellite Image Dataset for Ship Recognition and Some New Baselines. 2017.
Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, et al. ICDAR 2015 Competition on Robust Reading. Proceedings of the 2015 13th International Conference on Document Analysis and Recognition (ICDAR), 2015.
Vidit Jain, Erik Learned-Miller. FDDB: A Benchmark for Face Detection in Unconstrained Settings. 2010.
C P Papageorgiou, M Oren, T Poggio. A general framework for object detection. Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271), 1998.
Constantine Papageorgiou, Tomaso Poggio. A Trainable System for Object Detection. International Journal of Computer Vision, 2000.
Anuj Mohan, Constantine Papageorgiou, Tomaso Poggio. Example-Based Object Detection in Images by Components. IEEE Trans. Pattern Anal. Mach. Intell., 2001.
Paul Viola, Michael Jones. Rapid object detection using a boosted cascade of simple features. CVPR, 2001.
N Dalal, B Triggs. Histograms of oriented gradients for human detection. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), 2005.
David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 2004.
Pedro F Felzenszwalb, David A McAllester, Deva Ramanan. A discriminatively trained, multiscale, deformable part model. 2008 IEEE Conference on Computer Vision and Pattern Recognition, 2008.
P F Felzenszwalb, R B Girshick, D McAllester. Cascade object detection with deformable part models. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010.
T Malisiewicz, A Gupta, A A Efros. Ensemble of exemplar-SVMs for object detection and beyond. 2011 International Conference on Computer Vision, 2011.
Ross B Girshick, Pedro F Felzenszwalb, David A McAllester. Object Detection with Grammar Models. Advances in Neural Information Processing Systems 24, 2011.
Ross Brook Girshick. From Rigid Templates to Grammars: Object Detection with Structured Models. University of Chicago, 2012.
Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. Commun. ACM, 2017.
R Girshick, J Donahue, T Darrell, J Malik. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014.
K E A van de Sande, J R R Uijlings, T Gevers, A W M Smeulders. Segmentation as selective search for object recognition. 2011 International Conference on Computer Vision, 2011.
P F Felzenszwalb, R B Girshick, D McAllester, D Ramanan. Object Detection with Discriminatively Trained Part-Based Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010.
R B Girshick, P F Felzenszwalb, D McAllester. Discriminatively Trained Deformable Part Models, Release 5. http://people.cs.uchicago.edu/\~rbg/latent-release5/.
R Girshick. Fast R-CNN. 2015 IEEE International Conference on Computer Vision (ICCV), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition. Computer Vision – ECCV 2014, 2014.
Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Proc. of NeurIPS, 2015.
Matthew D Zeiler, Rob Fergus. Visualizing and Understanding Convolutional Networks. Computer Vision – ECCV 2014, 2014.
Jifeng Dai, Yi Li, Kaiming He, Jian Sun. R-FCN: Object Detection via Region-Based Fully Convolutional Networks. Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016.
Zeming Li, Chao Peng, Gang Yu, Xiangyu Zhang, Yangdong Deng, Jian Sun. Light-Head R-CNN: In Defense of Two-Stage Object Detector. 2017.
François Chollet. Xception: Deep Learning with Depthwise Separable Convolutions. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
T Lin, P Dollár, R Girshick, K He, B Hariharan, S Belongie. Feature Pyramid Networks for Object Detection. Proc. of CVPR, 2017.
J Redmon, S Divvala, R Girshick, A Farhadi. You Only Look Once: Unified, Real-Time Object Detection. Proc. of CVPR, 2016.
J Redmon, A Farhadi. YOLO9000: Better, Faster, Stronger. Proc. of CVPR, 2017.
Joseph Redmon, Ali Farhadi. YOLOv3: An Incremental Improvement. 2018.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E Reed, Cheng-Yang Fu, et al. SSD: Single Shot MultiBox Detector. Proc. of ECCV, 2016.
T Lin, P Goyal, R Girshick, K He, P Dollár. Focal Loss for Dense Object Detection. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2020.
A Shrivastava, A Gupta, R Girshick. Training Region-Based Object Detectors with Online Hard Example Mining. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
S Xie, R Girshick, P Dollár, Z Tu, K He. Aggregated Residual Transformations for Deep Neural Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Cheng-Yang Fu, Wei Liu, Ananth Ranga, Ambrish Tyagi, Alexander C Berg. DSSD : Deconvolutional Single Shot Detector. 2017.
Abhinav Shrivastava, Rahul Sukthankar, Jitendra Malik, Abhinav Gupta. Beyond Skip Connections: Top-Down Modulation for Object Detection. 2017.
Lichao Huang, Yi Yang, Yafeng Deng, Yinan Yu. DenseBox: Unifying Landmark Localization with End to End Object Detection. 2015.
Jiahui Yu, Yuning Jiang, Zhangyang Wang, Zhimin Cao, Thomas Huang. UnitBox: An Advanced Object Detection Network. Proc. of ACM International Conference on Multimedia, 2016.
Hei Law, Jia Deng. CornerNet: Detecting Objects as Paired Keypoints. International Journal of Computer Vision, 2019.
Alejandro Newell, Zhiao Huang, Jia Deng. Associative Embedding: End-to-End Learning for Joint Detection and Grouping. Advances in Neural Information Processing Systems 30, 2017.
K Duan, S Bai, L Xie, H Qi, Q Huang, Q Tian. CenterNet: Keypoint Triplets for Object Detection. Proc. of ICCV, 2019.
Xingyi Zhou, Jiacheng Zhuo, Philipp Krähenbühl. Bottom-up Object Detection by Grouping Extreme and Center Points. Proc. of CVPR, 2019.
Zhi Tian, Chunhua Shen, Hao Chen, Tong He. FCOS: Fully Convolutional One-Stage Object Detection. Proc. of ICCV, 2019.
Tao Kong, Fuchun Sun, Huaping Liu, Yuning Jiang, Jianbo Shi. FoveaBox: Beyond Anchor-based Object Detector. 2019.
K He, G Gkioxari, P Dollár, R Girshick. Mask R-CNN. Proc. of ICCV, 2017.
Liang-Chieh Chen, George Papandreou, Florian Schroff, Hartwig Adam. Rethinking Atrous Convolution for Semantic Image Segmentation. 2017.
Xue Yang, Junchi Yan, Xiaokang Yang, Jin Tang, Wenlong Liao, Tao He. SCRDet++: Detecting Small, Cluttered and Rotated Objects via Instance-Level Feature Denoising and Rotation Loss Smoothing. 2020.
J Ding, N Xue, Y Long, G Xia, Q Lu. Learning RoI Transformer for Oriented Object Detection in Aerial Images. Proc. of CVPR, 2019.
Wen Qian, Xue Yang, Silong Peng, Yue Guo, Chijun Yan, Junchi Yan. Learning Modulated Loss for Rotated Object Detection. Proc. of AAAI, 2019.
Peng Sun, Guang Chen, Guerdan Luke, Yi Shang. Salience Biased Loss for Object Detection in Aerial Images. 2018.
Jiaming Han, Jian Ding, Jie Li, Gui-Song Xia. Align Deep Features for Oriented Object Detection. IEEE Trans. on Geoscience and Remote Sensing, 2021.
J Dai, H Qi, Y Xiong, Y Li, G Zhang, H Hu, et al. Deformable Convolutional Networks. Proc. of ICCV, 2017.
Y Zhou, Q Ye, Q Qiu, J Jiao. Oriented Response Networks. Proc. of CVPR, 2017.
Xue Yang, Junchi Yan. Arbitrary-Oriented Object Detection with Circular Smooth Label. Proc. of ECCV, 2020.
Youtian Lin, Pengming Feng, Jian Guan. IENet: Interacting Embranchment One Stage Anchor Free Detector for Orientation Aerial Object Detection. 2019.
Zhifeng Xiao, Linjun Qian, Weiping Shao, Xiaowei Tan, Kai Wang. Axis Learning for Orientated Objects Detection in Aerial Images. Remote Sensing, 2020.
Zhiming Chen, Ke-An Chen, Weiyao Lin, John See, Hui Yu, Yan Ke, et al. PIoU Loss: Towards Accurate Oriented Object Detection in Complex Environments. Proc. of ECCV, 2020.
Lin Zhou, Haoran Wei, Hao Li, Wenzhe Zhao, Yi Zhang, Yue Zhang. Objects detection for remote sensing images based on polar coordinates. 2020.
Haoran Wei, Yue Zhang, Zhonghan Chang, Hao Li, Hongqi Wang, Xian Sun. Oriented objects as pairs of Middle Lines. ISPRS Journal of Photogrammetry and Remote Sensing, 2020.

http://host.robots.ox.ac.uk/pascal/VOC/, accessed 2021-04-10 ↩︎
http://image-net.org/challenges/LSVRC/, accessed 2021-04-10 ↩︎
http://cocodataset.org/, accessed 2021-04-10 ↩︎
https://storage.googleapis.com/openimages/web/index.html, accessed 2021-04-10 ↩︎
https://captain-whu.github.io/ODAI/, accessed 2021-04-10 ↩︎
https://www.kaggle.com/guofeng/hrsc2016, accessed 2021-04-10 ↩︎
https://iapr.org/archives/, accessed 2021-04-10 ↩︎
http://vis-www.cs.umass.edu/fddb/, accessed 2021-04-10 ↩︎

An arXiv PDF command-line interface downloader

Mon, 12 Jul 2021 00:00:00 +0000

The Hassle of Downloading arXiv Papers

A few months ago I had acquired a tablet to read and annotate research papers wherever I want, stopping the waste of printing these in paper form and then keeping the annotated printed versions somewhere in my desk. This led me to a setup in which I store all research papers in some directory structure at ~/papers/ and have this directory be in sync with a synchronization service; currently OneDrive, which has an excellent third-party Linux client.

Many of the papers I find are hosted on arXiv.org. When I’ve got a paper which I’d like to read, e.g. Variational Diffusion Models, I first have to manually download the paper into some location. By default, this results in a file named after the arXiv paper id, e.g. 2107.00630.pdf, which is annoying as the id is hard to associate with the actual paper title. Therefore, I usually rename the paper according to its title:

$ mv 2107.00630.pdf ~/papers/generative-models/2107.00630v1.Variational_Diffusion_Models.pdf

Now the paper is ready to be automatically synced and I can find it on my tablet device. After repeating the above multiple times, I was especially annoyed by the file renaming necessity. Furthermore, it also happens that I often simply have the arXiv link and know that I want to have this paper on my tablet. I’m an avid Linux user and work from the terminal most of the time. Hence, it was natural to write a command-line tool that takes an arXiv link/id, downloads it into a preferred directory, and automatically renames the filename according to the title. This is when I came up with a simple tool called arxiv-downloader, wrapping the neat arXiv Python wrapper lukasschwab/arxiv.py.

The Solution: arxiv-downloader

This little tool is available on PyPi (pip install arxiv-downloader) and offers the arxiv-downloader script in the command-line:

$ arxiv-downloader --url https://arxiv.org/abs/2107.00630 --directory ~/papers/generative-models/

Directory /home/steven/papers/generative-models/ does not exist. Create? [y/n] y
Starting download of article: "Variational Diffusion Models" (2107.00630)
Download finished! Result saved at:
/home/steven/papers/generative-models/2107.00630v1.Variational_Diffusion_Models.pdf

This merges the sequence of opening an arXiv link, manually downloading the PDF, renaming this PDF according to the paper title, and moving the file to the final location, into a single command. Although this might only save me about half a minute for every paper, I deeply felt the need to automate these steps. If you’re a computer scientist or programmer, I hope you can relate; if not then just call me crazy :-).

You can check out all available arguments with arxiv-downloader -h

$ arxiv-downloader -h
usage: arxiv-downloader [-h] [--url URL] [--id ID] [--directory DIRECTORY] [--source]

arXiv Paper Downloader.

optional arguments:
 -h, --help show this help message and exit
 --url URL, -u URL arXiv article URL.
 --id ID, -i ID arXiv article ID (for https://arxiv.org/abs/2004.13316 this would be
 2004.13316).
 --directory DIRECTORY, -d DIRECTORY
 Output directory.
 --source, -s Whether to download the source tar file.

The source code is available at braun-steven/arxiv-downloader. Feel free to contribute, leave feedback, and report issues.