Data flow is a recurrent use case faced by many companies and institutions. There is always the need to handle an incoming source of data, transform it somehow, and send it to another system. These data flows evolve quickly and are sometimes mission-critical.
Monitoring, changing, investigating, and fixing your ETL jobs could be challenging. Mostly if your ETLs are a bunch of scripts triggered via a CRON scheduler or when a new message arrives in your queue messaging system. Most of the time no user interface is available to visualize your Data flows to investigate what’s going on. In that…
It’s easy to write asynchronous Python code that does not behave like you think it would. I recently faced a situation where a Sanic web application was crashing/restarting in Kubernetes very often, but without any trace.
Why was it crashing/restarting without any logs? I was clueless. The only possible cause that came to my mind was the readiness/liveness probe that I configured on that application. Then, it would have meant that application was not able to answer health checks — at least not fast enough.
It could also be the main process that was stuck doing something, and not able…
This article will describe creation of two simple micro-services using Sanic with multiple workers, and Jaeger configuration that will help visualizing flows them.
From Sanic official Documentation :
“ Sanic is a Python 3.6+ web server and web framework that’s written to go fast. It allows the usage of the async/await syntax added in Python 3.5, which makes your code non-blocking and speedy.”
In fact, writing an API with Sanic is as simple as :
from sanic import Sanic
from sanic import response
import randomapp = Sanic(__name__)@app.route("/example-api") async def example(request): random_value = random.randint(0, 100000) return response.json(random_value) if __name__…
