Author name: cndro

In this tutorial, we will discuss what Docker Compose is all about. In one of our previous posts, we mentioned that Docker is a containerization platform developed to help package applications and their dependencies in the form of containers. This allows these applications to work effortlessly in any environment, either development, test, or Production. We also stated that Docker was introduced to tackle many problems developers face on staging servers and development. Here now, we will discuss how Docker Compose is majorly used. Docker compose helps to facilitate multiple services running simultaneously. We run multiple containers as a single service here, and each of these containers runs in isolation but communicate with each other when required. Installation Firstly, we need to install Docker Compose on our machine. To install Docker Compose on Linux, Windows, or Mac, you can visit this link. After installing it, verify with the command docker-compose –version on your terminal. How to Create a Compose File Hello World Docker Compose Let’s look at this basic example of  creating Hello World with Docker Compose. In the code below, we created a container from the ubuntu image, whereby the “Hello World” service is initialized from the ubuntu image, which will print “Hello World” when run. 1 version: ‘2’ 2 services: 3 hello_world: 4 image: ubuntu 5 command: [/bin/echo, ‘Hello world’] We use docker-compose up to run the code; Let’s try out another example of how much further we can use the Docker Compose for our application. Using Redis in Flask App We use Redis to handle our requests in this application, whereby we pass our data via Redis. What we have below is our requirements.txt file which is flask and Redis. Requirements.txt Our Flask App Here is the code to our flask app ,we’re using the POST and GET method. The POST method was used to send data which is in form of {‘name’: your_input} via the Redis as a collection of string keys and values. Whenever the GET method endpoint is called, we can get our items, which will be returned via the redis.lrange(). from flask import Flask, request, jsonify from redis import Redis app = Flask(__name__) redis = Redis(host=”redis”, db=0, socket_timeout=5, charset=”utf-8″, decode_responses=True) @app.route(‘/’, methods=[‘POST’, ‘GET’]) def index(): if request.method == ‘POST’: name = request.json[‘name’] redis.rpush(‘customers’, {‘name’: name}) return jsonify({‘name’: name}) if request.method == ‘GET’: return jsonify(redis.lrange(‘customers’, 0, -1)) The DockerFile This is our DockerFile which contains series of instructions on how we want our image to be created. FROM python:3.7.0-alpine3.8 COPY requirements.txt /app/ WORKDIR /app RUN pip install –no-cache-dir -r requirements.txt COPY . . ENV FLASK_APP=app.py CMD flask run –host=0.0.0.0 Docker Compose File Under our docker compose file, we tell docker how to build our image and we specify Flask development environment. So whenever we run the docker-compose up, our flask app will run based on the port we specified there as well. version: “3” services: app: build: . image: flask-redis:1.0 environment: – FLASK_ENV=development ports: – 5000:5000 redis: image: redis:4.0.11-alpine Running Docker Compose File We use the docker-compose up in our terminal to run our compose file, then we have our application running as seen below. We can move further to test with Postman. Project structure Your project structure should look like this

                                                                    Photo by Myriam Jessier on Unsplash In this tutorial, we’ll look at how we can use Regular Expressions in Tableau. In our last tutorial, we discussed how we can use Regular Expressions in Python. Just like we learnt earlier, Regular Expression is used for extracting data element from a string of data. The common application of Regular Expression is website and mobile apps. The use of Regex here is basically for input validation. Many applications can validate if what a user enters matches the back end’s required structure. An example is our email address, which the system checks to see if we typed it correctly. Now, let’s look at the Metacharacters classes which build up our knowledge foundation in Regex. Regular Expression Metacharacters Classes The Metacharacters are classes useful in defining rules to find a specific pattern in a string. These classes do have a special meaning in the REGEX Engine. Few examples of the Metacharacters classes are listed below. Now that we’ve discussed the Metacharacters classes, this gives us an overview of how to interpret or define our Regex expressions. In Tableau, we have four different REGEXP functions, and they are; REGEXP_EXTRACT (string, pattern) REGEXP_EXTRACT_NTH (string, pattern, index) REGEXP_MATCH (string, pattern) REGEXP_REPLACE (string, pattern, replacement) Let’s discuss how each can be used and their various examples. REGEXP_EXTRACT (string, pattern): The REGEXP_EXTRACT is used for searching a particular pattern within a string or substring, and then extract the data element which is our output. Example: We have students’ descriptions with their names, subjects, and ages in the table below. We’ll demonstrate how we can use REGEXP_EXTRACT to extract each student’s first and last name. Extract Last Name To extract their last name, we will use the code below. In this code, we observed that the pattern (\w+) is a rule from the metacharacters we explained earlier. This is why understanding the metacharacters helps a lot in writing our regex. 1 REGEXP_EXTRACT([Text], ‘(\w+)’) The function of the pattern (\w+) here tells us the parenthesis acts as our capture group, and the backslash describes the starting point to start with. Now the “w” is matching on a word character and alphanumeric values. The function of the “+ “ symbol means to extract all characters once and stop when it finds a next character which isn’t a word. This is similar to what we have after each last name is a comma. As seen below, we should have our desired text extracted with this code. Extract First Name Let’s extract our first name . We’ll also define the right pattern we need to extract our data correctly. We have the code we’ll use for this below. 1 REGEXP_EXTRACT([Text], ‘,\s+(\w+)’) In this code, we observed it’s quite different from what we used to extract the last name. The reason is where the first name is located. Look at this first string, “Cooper, Elizabeth is a student of Account, Age: 15”. Elizabeth, the first name is located after a comma and a space. This constraint has to be dealt with to extract our data correctly. So with our pattern (R EGEXP_EXTRACT([Text], ‘,\s+(\w+)’), we first have the “,” whereby it is to search for the comma. Also, the “\s+” is for searching for space, and the plus symbol is used for any double space. Then we use the (\w+) to extract our word. REGEXP_EXTRACT_NTH (string, pattern, index) This is used to search for a particular pattern within a string or substring, starting at the nth position in the string, and extracting the data element. Example: String= ‘abc 123’   To extract 123 out, we use the pattern ‘([a-z]+)\s+(\d+)’ and also specify the item index position REGEXP_EXTRACT_NTH(‘abc 123’, ‘([a-z]+)\s+(\d+)’, 2) = ‘123’ REGEXP_MATCH (string, pattern): The REGEXP_MATCH is used to look for a particular pattern within a string or substring and returns TRUE if we have an exact match. Example: In the table below, we want to search if the Student’s Details contains an Email Address. We will specify our pattern as seen below to check for all email address in the data. 1 REGEXP_MATCH 2 ( 3 [Student Details], 4 ‘([A-Za-Z0-9 ._% + -] + @ [A-Za-Z0-9 .-] + \. [Az] {2,4})’ 5) We should have this result table; REGEXP_REPLACE (string, pattern, replacement) This function is used for searching for a particular pattern within a string or substring and replacing that data element with a different data element. Example: In this table below we want to replace all this email address with a space. To handle this, we’ll use the code below, which has same pattern we used earlier to search for email addresses. Whenever it finds the email address in the text, it replaces it with a space. 1 ( 2 [Messy Data], 3 ‘([A-Za-Z0-9 ._% + -] + @ [A-Za-Z0-9 .-] + \. [Az] {2,4})’, 4 ” 5 )   We should have our output as seen below; Hope you enjoyed this post. Thanks for reading.

                                                                      Photo by Alex Chumak on Unsplash In this tutorial, we’ll be discussing Regular Expressions in Python. According to Wikipedia, Regular Expressions can be defined as a sequence of characters that specifies a search pattern in text. We can go further by saying they are patterns used for matching character combinations in strings. Python provided a module that supports the use of regex. This module is known as re. We can import it for use in our code. So, we’ll discuss what the module entails and how we can use it. Let’s look at different examples of regex in Python. In this example, we use the search function to check if our test string starts with “I” and ends with “texas.” Below is our  result; 1 Got the search. Before we look at more examples, let’s discuss the different RegEx FunctionS available in Python. RegEx Functions 1 import re 2 3 pattern = ‘^I.*texas$’ 4 test_string = ‘I am in texas’ 5 result = re.match(pattern, test_string) 6 7 if result: 8 print(“Got the search.”) 9 else: 10 print(“Search unsuccessful.”)   findall: The findall function can be used to return all non-overlapping matches of the pattern in our string of data. Here in this example, we find all digits we have in our text string. 1 import re 2 # The string of text where regular expression will be searched. 3 string_1 = “””Here are some cudtomer’s id, Mr Joseph: 396 4 Mr Jones: 457 5 Mrs Shane: 222 6 Mr Adams: 156 7 Miss Grace: 908″”” 8 # Setting the regular expression for finding digits in the string. 9 regex_1 = “(\d+)” 10 match_1 = re.findall(regex_1, string_1) 11 print(match_1)   We should have this as our result. 1 [‘396’, ‘457’, ‘222’, ‘156’, ‘908’]   Search: . The search function is used with the regex module to search if a particular pattern exists within a string. If the search comes out successful, it returns the match object, but if not, it returns none. Look at the example we have here; In this example, we will use the search function to look for “school” in our string pattern. 1 2import re 3 4s = ‘what school did you graduated from?’ 5 6match = re.search(r’school’, s) 7 8print(‘Start Index:’, match.start()) 9print(‘End Index:’, match.end())   It gave us the result below. This result tells us where the start index of school is and the end. This should match if we count that from the string. 1 Start Index: 5 2 End Index: 11 Split: In the split function, it splits the string based on the occurrence of the regex pattern we specified and then returned the list containing our substrings. An example is here below for us to understand this better. In this example, we split the string by the first two hyphens and the pattern sequence for handling hyphens or non-alphanumeric characters. 1 import re 2 3#the pattern sequence for hyphen or non alphanumeric chracter 4pattern = r’\W+’ 5 6# our targeted string 7string = “100-joe-01-10-2022” 8 9#we want to split the string by the first 2 hyphens 10txt_ = re.split(pattern, string, maxsplit=2) 11 12print(txt_)   Output 1 [‘100’, ‘joe’, ’01-10-2022′] Sub: The sub-function can be used with the regex module to replace multiple elements in a string and now return the new replaced string. Let’s see the example below; 1 import re 2 3 # Our Given String 4 s = “Debugging is very important when coding.” 5 6 # Performing the Sub() operation 7 out_1 = re.sub(‘a’, ‘x’, s) 8 out_2 = re.sub(‘[a,I]’,’x’,s) 9 out_3 = re.sub(‘very’,’not’,s) 10 11 # Print output 12 print(out_1) 13 print(out_2)   Output 1 Debugging is very importxnt when coding. 2 Debugging is not important when coding.   Now, if we observe, we used different patterns in each of the functions we mentioned earlier. This brings us to Meta Characters. Python RegEx Meta Characters The Meta Characters are very useful in defining rules to find the specific pattern we want in a string. These Meta Characters are listed below; And that’s about Regular Expressions in Python. Thanks for reading this article. See you in the next post.

Datamart is one of the newest addition to the Power BI component. It helps to bridge the gap between business users and IT. It provides an easy and no-code experience to ingest data from different data sources and perform ETL on the data using Power Query. After that, we can load the data to an Azure SQL database which is fully managed and doesn’t require tuning or optimization. Datamart also provides a single platform to carry out all these processes without needing an extra tool, so we have our Dataflow, Azure SQL Database, Power BI Dataset, and a Web UI all in one place. The people who use Datamart consist of  Data Analysts, Developers, and Business owners.   Let’s discuss the features and benefits of Datamart. Features of Datamart in Power BI a)The Datamart tool has an automated performance tuning and optimization whereby you don’t need to do the tuning yourself. b)It is also a 100% web-based tool that requires no extra software. c) Datamart has a native integration with Power BI and other Microsoft analytics software. d)It has a friendly user interface. It requires no coding experience to use it. e)DataMart is supported to use with SQL and other in-demand client tools.   Benefits of DataMart a)Datamart is very efficient in data ingestion and performing Extraction, transformation, and loading of data with SQL b)To use Datamart, you don’t have to be a programmer. c)Datamart also provides self-service users to carry out relational database analytics without the aid of a database administrator. d)Datamart enable Power BI users to build end-to-end solutions without dependencies on other tooling or IT teams. e) Datamart provides a centralized small to moderate data volume (approximately 100 GB)for self-service users. Comparison Between DataFlow and DataMart Remember we talked about Power BI DataFlow in our earlier tutorial? We mentioned it’s a data transformation component in Power BI with a Power Query process running in the cloud. It helps store data into CDM(Common Data Model) inside Azure Data Lake storage. Power BI uses these Dataflows to ingest data into our Datamarts. We use dataflows whenever we want to utilize our ETL logic. When discussing the features, we talked about Datamart. We mentioned it’s a fully managed database that enables us to store our data in a relational managed Azure SQL DB; we also said it’s a no-code visual query designer. So in DataFlow, we can’t browse tables, query, or explore without providing our dataset, while in Datamart, it is possible to sort, filter, and do simple aggregation through SQL expressions. We also have access to our data via the SQL endpoint. DataFlow is usually used whenever we want to build reusable and shareable data prep in Power BI.

                                                        Photo by Rubaitul Azad on Unsplash In this tutorial, we will demonstrate how we can install Airflow using Docker. Docker is an open platform for developing and running different applications. Docker enables us to separate our applications from infrastructure so we can deliver our software faster. We can also manage our infrastructures the same way we manage our applications. Now, let’s look at Airflow. Apache airflow is one of the brilliant tools utilized by many companies in defining & scheduling their complex data pipelines. We can programmatically schedule and monitor the workflow for our different jobs. This tool is widely used by data scientists, data engineers, software engineers, and many more. We’ll use a step by step process to show how to carry out this installation. Step1: Install Docker Engine The first stage of our installation is installing docker itself on our machine. We will check our computer using this command docker –version if we have docker installed. So if not, we will walk you through how to install it. Based on what we have on Docker’s official website, these are the following steps for installing docker; Install using the repository 1 sudo apt-get update 2 3 sudo apt-get install \ 4 ca-certificates \ 5 curl \ 6 gnupg \ 7 lsb-release   Add Docker’s official GPG key: 1 curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg –dearmor -o /usr/ share/keyrings/docker-archive-keyring.gpg   Set up a stable repository. You can add the nightly or test repository by using the word nightly or test in the command below; 1echo \ 2 “deb [arch=$(dpkg –print-architecture) signed-by=/usr/share/ keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \ 3 $(lsb_release -cs) stable” | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null   Now, we can install the latest version of Docker Engine, containerd, and Docker Compose by using the command below. 1sudo apt-get update 2 sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin   We can verify if Docker has been installed on our machine by running thehello-world image below. 1$ sudo docker run hello-world   So, we’ve installed docker on our machine. Let’s proceed to the next step. We must also note that we need docker-compose before proceeding to Airflow. Nice, it has been installed along with the Engine. Let’s move to the next step. Step2: Working with Vs Code Let’s open our Visual studio code or any other IDE we have and create a new folder for our project. We can name the folder airflow-docker. Now we need to download a docker-compose file that describes all the services required by Airflow. The Docker Compose file has already been made for us by the Airflow community. We can run this code below to download the file into our working directory. 1 curl -LfO ‘https://airflow.apache.org/docs/apache-airflow/2.3.0/docker-compose.yaml’   Now we should have the docker-compose YAML in our working directory. Let’s create more new folders for dags, plugins and logs. The Dag folder is where our python file will reside. You should have this as seen below. Whenever you open the docker-compose file, you should have something similar to this; Under the services, you can see the Postgres user and password named as airflow. We’ll use this as our username and password when logging into the webserver on the web browser. Step3: Export Environment Variables Here, we also need to export our environment variables to ensure the users and group permissions are the same as folders from host and folders in our container. Run the command below on our terminal in VScode; 1echo -e “AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0” > .env After you run the command, you should have a .env file in your directory. Step4: Initialize Airflow Instance Now that we are done with all our settings, we can initialize our airflow instance using the command below. This will create both user and password airflow based on the settings in the docker compose file. 1 docker-compose up airflow-init   We should have something like this as shown below after we performed our initialization; The next thing is to run all the services we specified in the docker compose file(the redis, scheduler, worker, webserver, e.t.c) so that our containers will come up and start running. We will use the command below; 1 docker-compose up If you have any error relating to permission issue, ensure before each command you add Sudo at the back. This is because you haven’t added docker among groups. Now, we can check the browser to see our Airflow Instance currently running by using the localhost:8080 command to view it in the web browser. We should have a page like this whenever we type the command in our browser.

Today’s article focuses on Data joining in Tableau. In Tableau, when we work with enormous data sets, it is common to find that those data sets have multiple tables with different data fields. This tells us data usually don’t reside in a single table. We can have many different tables. It means we can join tables using columns that are common or related. These related fields are usually known as Key fields or records. The method of combining this data is referred to as Data Joining. We can combine tables based on the same or different data sources to create a single table. Let’s look at the different types of Joins we have in Tableau. Types Of Joins Inner Join Left Join Right Join Full Outer Join We’ll look at each of these joins and demonstrate with the Sample superstore dataset in our Tableau Desktop. Inner Join What is an inner data join? Inner Join in Tableau refers to how we can create a new table from two tables containing only values or columns common between these two tables. Let’s look at our demonstration below in Tableau Desktop. We will bring in our Superstore data, load it in the Data pane, and then do an inner join using the order ID from the Orders table and the OrderID Returns table. Let’s see the image below. Left Join Left join tableau is also a way of creating a new table that is formed from all data from the left table and only matching values from the right table. If there are no common rows in the right table, null values are returned in the new table. Let’s demonstrate this join in Tableau. We’ll still use the Orders table and Returns table in our superstore dataset. The image below shows how we did that. In the data view below, let’s also observe null data present in some of our columns. Right Join Right Join is formed in Tableau between two tables, whereby the resulting table or new table created contains all values of the right table and only matching values from the left table. For non-common rows, null values are returned. Look at the screenshot below to see our Right Data Join in Tableau. Full Outer Join The full outer join is a type of join formed between two tables, whereby the resulting table contains all the data values from both the left and right tables. Also, values that do not find a match in both tables are shown as null. A screenshot of how to perform a Full Outer Join is shown below; We hope you’ve been able to learn about Data joining in Tableau and types of joins. Thanks for reading