Wsrtjtyk

I am learning cv2 and trying to detect the on-board number tiles (dynamically) from the 2048 game, and outlining them in green.

Firstly I am having trouble detecting the ones in the more orange to red range (8, 16, 32, 64) and if I lower the threshold the whole board seems to be included. Sometimes, smaller parts (such as the round part of a 6 are included) or an entire tile is ignored. How would I go about detecting the tiles on a board like this?

Here is the code I have so far:

import cv2

import mss

import time

import numpy as np



# Static screenshot for board

monitor = {"top": 135, "left": 425, "width": 500, "height": 500}

sct = mss.mss()



# Run for a maximum of 150s or until 'q' is pressed

last_time = time.time()

while time.time() - last_time < 150:

    img = np.asarray(sct.grab(monitor))

    resized_img = cv2.resize(img, (100, 100))

    gray_img = cv2.cvtColor(resized_img, cv2.COLOR_BGR2GRAY)

    thresh = cv2.threshold(gray_img, 200, 255, 0)[1]

    contours = cv2.findContours(thresh, 1, 2)[1]

    for cnt in contours:

        if len(cnt) == 4:

            cv2.drawContours(resized_img, [cnt], 0, (0, 255, 0), 2)

    cv2.imshow("2048", resized_img)



    if cv2.waitKey(25) & 0xFF == ord("q"):

        break



cv2.destroyAllWindows()

sample detection:

EDIT: added sample input as requested

sample input:

sample output:

Thankyou for any responses, even if they are a point in the right direction

Rather than doing thresholding with the greyscale image, you can do thresholding on the color image with cv2.inRange. You can set the upper and lower bounds of allowed colors to include the numbered tiles but exclude the empty tiles and edges.

Also, I assume the step where you are checking if len(cnt) == 4: is to return only the square contours. However, resizing can result in contours that aren't exactly square for the tiles, and won't pass this check. Instead, you can get the outer contours of the tiles by changing the second input of findContours to 0 (contours = cv2.findContours(thresh, 0, 2)[1]) which sets the retrieval mode to cv2.RETR_EXTERNAL.

Here is the code with changes made, and appropriate upper and lower color bounds for the example image you gave.

import cv2

import mss

import time

import numpy as np



# Static screenshot for board

monitor = {"top": 135, "left": 425, "width": 500, "height": 500}

sct = mss.mss()



# inRange bounds

lower_bound = (0, 0, 210)

upper_bound = (230, 240, 250)



# Run for a maximum of 150s or until 'q' is pressed

last_time = time.time()

while time.time() - last_time < 150:

    img = np.asarray(sct.grab(monitor))[:,:,:3]

    resized_img = cv2.resize(img, (100, 100))

    mask = cv2.inRange(resized_img, lower_bound, upper_bound)

    contours = cv2.findContours(mask, 0, 2)[1]

    for cnt in contours:

        cv2.drawContours(resized_img, [cnt], 0, (0, 255, 0), 2)

    cv2.imshow("2048", resized_img)



    if cv2.waitKey(25) & 0xFF == ord("q"):

        break



cv2.destroyAllWindows()

Here is the output image created:

Edit: Here is the code for using the example image directly:

import cv2

import numpy as np



img = cv2.imread('2048.jpg')

resized_img = cv2.resize(img, (100, 100))

lower_bound = (0,0,210)

upper_bound = (230,240,250)

mask = cv2.inRange(resized_img, lower_bound, upper_bound)



contours = cv2.findContours(mask, 0, 2)[1]

for cnt in contours:

    cv2.drawContours(resized_img, [cnt], 0, (0, 255, 0), 2)

cv2.imshow('2048', resized_img)

cv2.waitKey(0)

One approach that you can take is to take the diff of this frame, from a snap-shot where all tiles are empty. This would get you the required mask with minimal computations.

Since you have not mentioned in the question that you have access to all empty tile snapshot, I would also present another technique known as color segmentation. Since the background color is consistent, but the numbered color tiles changes, So we will segment out the background first and then invert the mask to find contours as :

import cv2

import numpy as np





def threshold_tiles(board_img):

    board_bgd_color_low = np.array([155, 170, 140])

    board_bgd_color_high = np.array([200, 185, 195])



    board_empty_low = np.array([175, 180, 200])

    board_empty_high = np.array([185, 195, 210])



    mask_bgd = cv2.inRange(board_img, board_bgd_color_low, board_bgd_color_high)

    mask_tile = cv2.inRange(board_img, board_empty_low, board_empty_high)



    mask = cv2.max(mask_bgd, mask_tile)



    kernel = np.ones((7, 7), np.uint8)

    mask = cv2.dilate(mask, kernel)



    return ~mask





def get_box_contours(mask):

    _, cnt, hierarchy = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)



    cnt = filter(lambda x:cv2.contourArea(x) > 100, cnt)



    return cnt





def main():

    game_snapshot = cv2.imread("/path/to/img.jpg")



    # Crop the white borders

    game_snapshot = game_snapshot[5:-5, 5:-5]



    mask = threshold_tiles(game_snapshot)

    contours = get_box_contours(mask)



    for i in xrange(len(contours)):

        cv2.drawContours(game_snapshot, contours, i, (0, 255, 0), 3)



    cv2.imwrite("output.png", game_snapshot)





if __name__ == "__main__":

    main()

Intermediate masks:

+ =

Final mask for contours:

Output: