Categories
elixir

Code Evaluation/Testing Service using Docker API

Docker has been a game changer and has revolutionized containerization technology.

We are aware of the benefits of docker and how it helps us to run light weight containers easily without having to go through the hassles of using heavy virtual machines.

Most of us use the docker client to talk to the docker daemon, while this works for most of our use cases, did you know that you can also issue docker commands programmatically using the Docker API.

Source: https://docs.docker.com/get-started/overview/#docker-engine

Docker provides an API to interact with the docker daemon, as well as SDKs for Go and Python.

The Docker Engine API is a RESTful API accessed by an HTTP client such as wget or curl, or the HTTP library which is part of most modern programming languages.

So, if you want to issue commands to the docker daemon, you can do so programmatically using the API.

For example, to get a list of all running containers you can try…

curl --unix-socket /var/run/docker.sock http:/v1.24/containers/json

This will give us a list of all containers with various information in json format.

Use Case?

So, what can be the use cases for the Docker API, well if in any case you want to issue commands to the docker engine programmatically this API can help. 

One such use case could be running arbitrary code entered by the user in an isolated environment via docker.

Example:

Suppose you want to make a code evaluation service for a coding website. People can upload their code and this service will run and test the code against some test cases. However a person can upload some malicious code which can be potentially harmful for the system. 

To avoid this we will run the code inside docker containers to safely isolate and safeguard our servers from any potentially malicious code.

An Example to run arbitrary code inside docker container using the Docker API in elixir

The following is an example of how you can easily build a service like this in elixir, 

We will use HTTPPoison which is a HTTP client for Elixir to request the docker API.

Set the base url for request the docker API:

 # Base URL for docker unix socket  unix:///var/run/docker.sock

@base "http+unix://%2Fvar%2Frun%2Fdocker.sock"

Create a container, this will return the id of the container:

%HTTPoison.Response{body: body} =
  HTTPoison.post!(
    "#{@base}/containers/create",
      %{
       "Image" => image_name,
       "Cmd" => ["bin/bash", "-c", "tail -f /dev/null"], # keeps the container running
      },
    [{"content-type", "application/json"}]
  )

Start your container that you have created using its id:

HTTPoison.post!(
  "#{@base}/containers/#{container_id}/start",
  "",
  [],
)

Create an exec instance:

Here command is any command that you want to execute inside the container

For example:

The following json command string compiles a c code:

{\"AttachStderr\":true,\"AttachStdin\":true,\"AttachStdout\":true,\"Cmd\":[\"bin/bash\",\"-c\",\"gcc -std=c99 -o a.out main.c\"],\"Tty\":true}
HTTPoison.post!(
  "#{@base}/containers/#{container_id}/exec",
  command,
  [{"content-type", "application/json"}]
)

Start the execution of instance you have created

HTTPoison.post!(
  "#{@base}/exec/#{exec_id}/start?stream=1?stdout=1,stderr=1",
  "{\"Detach\":false,\"Tty\":true}",
  [{"content-type", "application/json"}]
)

Stop the container

HTTPoison.post!(
  "#{@base}/containers/#{container_id}/stop",
  "",
  []
)

Delete the container

HTTPoison.delete!(
  "#{@base}/containers/#{container_id}",
  []
)

The above example demonstrates how you can easily spin up docker containers to run arbitrary code safely isolated from your server.

Using this docker api you can run almost anything that you can using the docker client.

I hope this blog helps you someday if you encounter a possible use case for it.

Good Day ! 😃

Categories
elixir

csv2sql – Blazing fast Elixir Csv 2 Sql loader

Loading csv files to a database has always been a challenging task, without the database tables already created one has to go through the tiring process of manually inspecting the CSVs and creating the DDL queries to create the tables before the data can be loaded. 

Again, when dealing with huge csv files, the process might take a very long time to finish.

This is where csv2sql come into the picture.

Csv2sql is a blazing fast tool written in Elixir for loading csv files into the database.

Csv2sql can automatically infer the basic types of data stored in your csv file and create the database tables and load the data all with one single command.

Got a directory full of large csvs that needs to be loaded to a database?

With csv2sql it’s as simple as running a simple command like..

./csv2sql --source-csv-directory "/home/user/Desktop/csvs" --db-connection-string "username:password@localhost/my_database"

That’s all ! Csv2Sql will analyze the files and insert them in to inferred data tables.

After it is done, it will give you a summary of the results.

Is it fast ?

Yes, csv2sql is made using elixir which makes the most of your cpu power by processing the csvs parallely thus greatly reducing the overall time taken, so it’s perfect for huge csv files.

Dependencies ?

Csv2sql only require erlang to be installed, that’s it. 

Find the csv2sql executable here.

Finally if you want to know how it works or if you are interested in elixir or genservers, find the repository here, also this project is open source and all your contributions are welcome.

Good Day !