Before this case, take a look at the CPL library in Docker to see an introduction to running the MPI processes inside a single Docker container. However, a far more interesting application is to create two containers and then link the processes running in them with MPI sending information between them. First we need to get the CPL library docker image: 

sudo docker pull cpllibrary/cpl-library

Next, we want to start a coupled case with a persistent shared volume The shared volume:

  1. Will contain all the output of the run
  2. Can allow coupling between two docker containers

To starts an interactive bash session called cplrun with volume cpl-vol mounted under folder /cplruns 

sudo docker run -it --name cplrun1 --mount source=cpl-vol,target=/cplruns cpllibrary/cpl-library

In the first docker container, copy an example to cplruns 

rsync -avP /cpl-library/examples/minimal_send_recv_mocks/ /cplruns/

Change directory to cplruns and run the CFD example 

cd /cplruns/
mpiexec -n 1 python ./minimal_CFD.py

This will create a PORT file in /cplruns/ which will be used to link to another executable running in a second container. To understand how this work, check out the discussion in Run CPL library which outlines the use of MPI PORT as a way to avoid patching the two coupled codes. As a result, we need to open A NEW TERMINAL and create a second docker container running in this but with access to the same file system 

sudo docker run -it --name cplrun2 --mount source=cpl-vol,target=/cplruns cpllibrary/cpl-library

In this container, we change to the cplruns directory and simply start the corresponding MD run we want to couple, 

cd /cplruns/
mpiexec -n 8 python ./minimal_MD.py

The output from the coupled codes should appear in both terminals as they exchange information. This has only been tested on a single machine, ssh keys and a shared file system would need to be established between computers.

To see the location of the volume on the host system 

sudo docker volume inspect cpl-vol


Coupling OpenFOAM and LAMMPS

We can use this approach to couple OpenFOAM and LAMMPS. In this case, both Docker containers are built FROM the same cpllibrary/cpl-library so have identical MPI and CPL library. As before, open two separate terminals as we'll be creating a Docker container for each program.

Starting with the OpenFOAM docker container in terminal 1, 

sudo docker run -it --name cplopenfoam --mount source=cpl-vol,target=/cplruns cpllibrary/cplopenfoam

Now, we need to copy as example to the directory shared between containers, we'll take the granular example which uses an fcc lattice with fixed particles, 

rsync -avP /cpl-library/examples/fcc_fixedParticles /cplruns/
cd /cplruns/fcc_fixedParticles

To setup OpenFOAM, check out the contents of run.sh which does the setup on the mesh in parallel and runs the case. 

cd openfoam
blockMesh
decomposePar
cd ../
mpiexec -n 1 CPLSediFOAM -case openfoam/

This will wait after writing a portfile which allows connection when read by the corresponding LAMMPS instance that want to be coupled, 

Only CFD realm present in MPI_COMM_WORLD
opened port: tag#0$description#9b5483f241c$port#41258$ifname#172.21.0.2$ Attempting to write to file:./port
Portname written to file ./port

Next, in the other terminal start a LAMMPS Docker container, 

sudo docker run -it --name cpllammps --mount source=cpl-vol,target=/cplruns cpllibrary/cpllammps

In the container, we need to change to the run directory cplruns and run, 

cd /cplruns
mpiexec -n 1 lmp_cpl < lammps/fcc.in

the LAMMPS instance runs, links to OpenFOAM in a different Docker container with communication through the MPI port system and runs a case.