Process multiple lines of unstructured logs and combine them as one structured record prior stashing them

Prologue

Log centralisation plays a key role in modern application support, analysis, and monitoring. ELK (Elasticsearch, Logstash, Kibana) stack has become one of the trending monitoring tool stacks that caters log centralisation use case. In such a setup Logstash is typically the one to receive log data for collecting, parsing, and transforming them into structured and meaningful data prior ingesting them to Elasticsearch for stashing.

Logstash, in my opinion, is equipped with a toolset to process any kind of machine generated unstructured data into stash-able structured data. …


Overview

‘If you are not a contributor to the Python Source Code then you would never need to build Python from the source’. This is exactly what I was thinking until I joined the IT department of a bank in Singapore as a Software Engineer. However, the situation that made me want to build Python from source is not important but the issues that I faced during doing so might be of interest for whoever trying to do the same.

In modern application development web-requests are inevitable. For the security of those requests HTTPS is used and HTTPS depends on TLS (Transport Layer Security). SSL (Secure Socket Layer) provides the functionality required by TLS. …


Overview

Problem Statement

A few days back I started working in the IT department of a bank as a Senior Software Engineer. This is the first experience with a significant restriction to the internet connectivity for both my workstation and for the servers where we are supposed to deploy the developed Python applications.

No internet connectivity at all for the servers and restricted internet connectivity for the workstations. We could only download files from the internet but this goes through a security scan before allowing us to download the file. …


As a beginner I thought the concept behind docker is similar to the concept behind Java, where you build once and deploy anywhere. For Java this capability is achieved through JVM (Java Virtual Machine). We need to install the corresponding compatible JVM for our Operating System. This JVM provides the isolation layer for Java programs to run on any platform (i.e on any OS). However, docker does not provide such isolation for docker containers. Windows based containers should run on Windows hosts, Linux based containers should run on Linux hosts.

Image for post
Image for post

Windows have introduced Hyper-V isolation to run Linux containers on Windows. However, this is again similar to Windows running a Linux VM and on top of that run Linux docker containers. At the time of writing this is experimental and does not support on Windows Server because it is not recommended in production deployments. …


Flask is a WSGI (Web Server Gateway Interface) compatible micro web framework written in Python. The easy learning curve and wide adoption have made Flask, one of the first choices for REST-API development in python. Due to the wide adoption, we can find a lot of tutorials and blog posts on ‘How to Deploy Flask Apps in Linux Servers’ in contrast to ‘Flask App Deployment in Windows’. However, there can be situations where one want to deploy one’s Flask applications on Windows-Servers too. In such a scenario someone might find this blog post handy and helpful.

Prerequisites

Before starting the guide you need to make sure that your Windows machine has following installed. …


MySQL is one of the prominent open source RDBMSs. In the cases of traffic growth to DBMS and when in the cases of backing up data replication is there for the rescue. MySQL comes with a set of in-built replication mechanisms that can be utilized as the need arise. Mainly there are 2 replication setups, master-master replication, and master-slave replication. As the names imply master-master is used as a load balancing technique in the presence of high write load. In contrast, master-slave is utilized when read load is significantly high, and for data backup purposes. …


Debugging is inevitable when you are not a brilliant programmer. I’m a terrible programmer and debugging is the only way for survival:). Even though debugging flask applications with PyCharm IDE is as simple as bread and butter, tend to forget it time to time, thus thought of making note of it.

if __name__ == '__main__':
app.run()

Insert the above in any place you’d like. Make sure you have done the imports right. Here to run() method you can give whatever the parameters you desire.

Then, right-click on the script that you have put the above code snippet and click on debug. Add debug points wherever you want and happy debugging. Hola!

For people who know python and PyCharm, yep, you got it right :), that’s how you debug a python script with PyCharm.


Once in a while even for Linux/Ubuntu servers you might want remote desktop access. Yeah, you heard right! It is graphical user interface access to the operating system. It is true that we can do anything with Linux/Ubuntu operating systems with full privileged shell access. However, there can be some software applications that at some point of time might need actual GUI interactions to proceed. If you are at such a situation this blog post is for you.

Image for post
Image for post

Recently as per the instructions of team lead had to install ‘Wolfram Mathematica 11.3.0’ in one of our Ubuntu 16.04 instances at Google Cloud. After installation to license activation had no other choice than figuring out a way to get remote desktop access so that can proceed with license activation process. …


At the time of this writing I have been coding in python for five straight months. I came from Java/Javascript background and I was frustrated and always complaining about python packaging system. It was mainly because I did not understand the simple but effective concepts behind python packaging and also because I was thinking in terms of Java/Javascript. Now I vouch for python, python packaging, and pythonic way.

Without further ado, let’s get back to our topic. Python packaging totally based on the number of directories you have within a python project. …


Often python packages need to depend on files which are not python source (.py) files such as images, data tables, documentation, and etc. Those files need special treatment in order for setuptools to handle them correctly.

The mechanism that provides this is the MANIFEST.in file. This is relatively quite simple. MANIFEST.in is really just a list of relative file paths specifying files or globs to be included while packaging.

include README.rst
include docs/*.txt
include funniest/data.json

In order for these files to be copied at install time to the package’s folder inside site-packages, you’ll need to supply include_package_data=True to the setup() function. …

About

Thilina Madumal

Senior Software Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store