Title: ctypes, OpenCV and Pillow
Slug: ctypes-opencv-pillow
Date: 2021-04-25 18:10:00
Author: Kartones
Lang: en
Tags: Development, Python, Tools, Graphics, Testing, Game Hacks
og_image: https://images.kartones.net/posts/kartonesblog/opencv_clicker.jpg
Description: Using Python libraries ctypes, OpenCV, and Pillow to automate tasks in video games, such as creating an auto-clicker, with challenges in handling input and item detection.


Note: This is a work in progress, consider this post more like some development notes of sources and links and references that I've found interesting. I might throw away half of it, or never finish the experiment.

What do [ctypes](https://docs.python.org/3/library/ctypes.html), [OpenCV-python](https://pypi.org/project/opencv-python/) and [Pillow](https://pypi.org/project/Pillow/) have in common, apart from being Python libraries? In my case they form a nice pack to begin tinkering with automation in less usual scenarios... like videogames.

After finding an amazing tutorial on [how to teach an AI to drive cars in GTA V with Python](https://pythonprogramming.net/game-frames-open-cv-python-plays-gta-v/), I thought I could start maybe with something simpler. Like an auto-clicker for those idle & clicker videogames.

I am using Pillow because its [ImageGrab](https://pillow.readthedocs.io/en/stable/reference/ImageGrab.html) module is perfect to capture screenshots (with bounding box directly, so no need to crop anything), and it's a single line of code to transform the image into an optimized [NumPy](https://numpy.org/) array... which is also the source for many OpenCV operations.

Regarding OpenCV, I'm just learning it as I go, half by reading [its tutorials](https://docs.opencv.org/master/d6/d00/tutorial_py_root.html) and half by directly playing with things I've seen in the GTA V AI article, like [changing colorspaces](http://web.archive.org/web/20210507012357/https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_colorspaces/py_colorspaces.html) and [canny edge detection](http://web.archive.org/web/20210301012650/https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_canny/py_canny.html). I am still toying with which approach is better (grayscale or HSV), but I already have a working prototype that draws edges of coins as a black & white mask, and then by reading pixel data I'm able to pinpoint them in screen.

Handling input is becoming actually the hardest part, as I want to build it for Linux (there are lots of articles on `ctypes` and Windows DirectInput) and I'm not familiar with the [X Window System](https://www.x.org/). Thankfully the [X11 library documentation ](https://www.x.org/archive/X11R6.8.1/doc/manindex3.html) contains all calls and I already have implemented querying the mouse position ([XQueryPointer](https://www.x.org/archive/X11R6.8.1/doc/XQueryPointer.3.html)) and moving it to wherever I want ([XWarpPointer](https://www.x.org/archive/X11R6.8.1/doc/XWarpPointer.3.html)), so I see where it detects entities better than just raw X,Y coordinates.

I am missing how to send mouse clicks, and I'd like to play with OpenCV a bit more to try to really detect items within it (right now is raw array comparisons).

I don't know if to publish the code, partly to not ease cheating on those games (which in the end you play for free) and party because excepting the input handling code, everything else are really basic operations with each library that you can find on their respective official documentation or tutorials.

In any case, here's one screenshot of a debug frame mask with edge detection. The game was running and already past the equivalent frame, but the goal is to automatically click to pick up all gold coins enemies leave upon dying:

![A clicker game and OpenCV edge mask frame](https://images.kartones.net/posts/kartonesblog/opencv_clicker.jpg)