# Image Processing, Etch-A-Snap

Image Processing

#### Etch-a-Snap Series

Etch-A-Sketch is a very simple 2D plotter which is limited to drawing a single unbroken line, of a single thickness and colour which is dark-ish on a silver-ish screen. To be able to plot a photo onto the Etch-A-Sketch we need to transform the image so that —

1. the resolution makes sense for movements on the screen
2. it is 1-bit, on/off only (no grayscale here)
3. it can be drawn in one continuous line

## Screen resolution

Once I had the basic control of the stepper motors sorted out the first step was to determine the resolution by drawing series of up-right-down-right-up-right-down-right patterns across the screen. By increasing and decreasing the length of the right portions I found the step size which when drawn drew what looked like two adjacent lines.

The resulting plotter scale was `25` — that is to step right by one pixel, we need to rotate the wheel 25 steps1. By drawing around the screen a max resolution of (240x144) was calculated for this scale.

The screen has a ratio of 5:3 so any other similarly scaled resolution will also work. Some examples below use a lower resolution of 100x60.

The input image is resized by scaling the dimension (width/height) which is closest to the target size, ensuring we don’t shrink the image smaller than the target in either dimension. The resulting image is then cropped to size.

python
``````from PIL import Image

t_width, t_height = (240, 144)
c_width, c_height = image.size

wr, hr = t_width / c_width, t_height / c_height

# Scale the dimension closest to the target.
ratio = wr if wr > hr else hr
target = int(c_width * ratio), int(c_height * ratio)

image_r = image.resize(target, Image.ANTIALIAS)

c_width, c_height = image_r.size

# Crop a rectangular region from this image.
# The box is a 4-tuple defining the left, upper, right, and lower pixel coordinate.
hw, hh = c_width // 2, c_height // 2
x, y = t_width // 2, t_height // 2

image_r = image_r.crop((hw - x, hh -y, hw + x, hh + y))
``````

This gives us an image at the target size of 240x144, with no blank space.

We capture from the camera at a 800x600 resolution.

## Finding some lines to draw

Before we do anything else we need something to draw. Because of the limitations of an Etch-A-Sketch that thing needs to be 1 bit (on/off) and preferably line-like.

The first thing I tested was the edge-enhance algorithms available in `pillow` since this was already a dependency. Unfortunately while it highlights edges, it doesn't have any noise-limiting and also produces regions of solid black. While these aren't impossible to draw with the Etch-A-Sketch they ain't pretty.

The best results were found from using the Canny Edge Detection algorithm in available in the Python computer vision library `cv2`.

This adds another dependency (and one not particulary easy to install) but it picks out the edges nicely, producing a series of 1 pixel thick lines from an input image. The code to extract the edges from an image is shown below.

python
``````from PIL import ImageOps
import cv2
import numpy as np

gray = image_r.convert('L')

ocv = np.array(gray)
threshold1 = 200
threshold2 = 50
edgec = cv2.Canny(ocv, threshold1, threshold2)
edgec = Image.fromarray(edgec)
edgec = ImageOps.invert(edgec)
edgec
``````

The values are the max and minimum threshold for local intensity gradient (change in image intensity) for something to be considered an edge. If something is above the max value `threshold1` then it is considered an edge, if it is below `threshold2` then it definitely isn't an edge. If it is between `threshold1` and `threshold2` it might be an edge, and whether it is is determined by whether it is connected to a pixel that is an edge.

But, to be completely honest, the above values above were determined through trial and error. They are pretty well tuned for both the type of input and the output scale.

If you want to experiment `200` & `100` also works well, producing sometimes neater images at the expense of detail/connectivity.

At this point we have a bunch of lines. What we need next, in order to be able to draw the lines on an Etch-A-Sketch, is to connect them all up.

Over 10,000 developers have bought Create GUI Applications with Python & Qt!

To support developers in [[ countryRegion ]] I give a [[ localizedDiscount[couponCode] ]]% discount on all books and courses.

[[ activeDiscount.description ]] I'm giving a [[ activeDiscount.discount ]]% discount on all books and courses.

## Continuous line drawing

To draw the entire picture with a single continuous line all regions of the image must be connected. I implemented two ways to do this, first using hatch-filling to increase the chance of lines connectin regions, and another which simply draws connecting lines.

The first of these happens at the image processing stage and is described below, the second occurs during graph-optimisation and is described in the next part.

python
``````fill_patterns = [
np.array([[1]]),
1-np.eye(16),
1-np.eye(8),
1-np.eye(4),
]
``````

The input image, in grayscale, is lowered to 4 levels of brightness. These 4 levels of grey (from white to black) are then replaced with a different densities of hashed lines, producing the darkening effect on the screen, and hopefully connecting up some additional regions.

python
``````def line_fill(img, mask_expand=0):
data = np.array(img)   # "data" is a height x width x 4 numpy array
output = data.copy()

width, height = data.shape

for n, pattern in enumerate(fill_patterns):

p_width, p_height = pattern.shape

fill_image = np.tile(pattern * 255, (width // p_width + 1, height // p_height + 1))
fill_image = fill_image[:width, :height]  # Drop down to image dimensions, so we map straight through.

return Image.fromarray(output)
``````

The resulting image should have the majority of the elements in the view connected and drawable by the Etch-A-Sketch. Regions of darkness in the input image are shaded, adding a bit of depth to the picture.

The downside with this shader-fill approach is that it creates a lot of extra detail to draw and can make busy pictures very noisy. An alternative which simply draws linker-lines between adjacent regions was also implemented, and is covered in the next section.

You can adjust the levels of brightness/shading by adding more entries to the `fill_pattern` list. Hatch patterns also work, although the Etch-A-Sketch draws them even worse.

### Further optimisations

Placing hashed lines over regions of brightness doesn't help in those cases where there are isolated regions of darkness in the middle of an image, surrounded by light. To try and solve this the shader areas were also expanded to increase the chance of overlap. As we apply the darker areas after the lighter areas, the effect is to increase dark hatching.

python
``````from PIL import ImageFilter

yd, xd = data.shape
output = data.copy()

for _ in range(iters):
for y in range(yd):
for x in range(xd):
if (
(y > 0 and data[y-1,x]) or
(y < yd - 1 and data[y+1,x]) or
(x > 0 and data[y,x-1]) or
(x < xd - 1 and data[y,x+1])
):
output[y,x] = True
data = output.copy()
return output

lined = line_fill(grayi, 4)
lined
``````

The downside of this algorithm is that it can produce dangling edges over the edges of lines, which causes yet more noise (and stuff to draw). Once the linking-lines algorithm (later) was added, this optimisation became redundant and was removed.

## What's next

We now have a processed 1 bit image, with (hopefully) the majority of regions in the image connected and able to be drawn in one continuous line. The next step is to work out how to draw this to the screen.

1. there is some give in the wheels, so when changing direction we also need to apply tracking corrections.

Continue reading the Etch-a-Snap series with Drawing, Etch-A-Snap