Here we will do an introduction to dsco commands.
To do this we will be setting up a new project called foo.
[1]:
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:85% !important; }</style>"))
Create a working directory:
[2]:
!mkdir -p /tmp/dsco_overview
[3]:
cd /tmp/dsco_overview/
/private/tmp/dsco_overview
We can use --help to see a list of the available commands and a short description of what they do.
[4]:
!dsco --help
usage: dsco [-h]
{build,code,documentation,go,init,ls,reports,restart,rm,rmi,shell,up,update_port}
...
positional arguments:
{build,code,documentation,go,init,ls,reports,restart,rm,rmi,shell,up,update_port}
available functionality
build build images. No flag builds dev.
code launch vs-code
documentation view and update documentation in the dev container
go Build containers, then open the web page interface.
init create a new project
ls list containers
reports view and update reports in the dev container
restart restart nginx and flask (dev)
rm remove containers
rmi remove images
shell start shell in container
up create and launch containers
update_port update container port (container restart required)
optional arguments:
-h, --help show this help message and exit
[5]:
!dsco init --help
usage: dsco init [-h]
optional arguments:
-h, --help show this help message and exit
The first thing we’ll want to do is create our project. We do this with --init
Normally the --init command prompts us for input. However this is being done in a jupyter notebook, which does not allow for interactive responses. To get around this we will pipe in our responses as follows:
- project_name [MyProject]: foo
- author [Discovery]:
- version [0.1]:
- year [2020]:
- project_port[8001]:
- select python_version:
1 - 3.7
2 - 3.6
Choose from 1, 2 [1]:
The only default we’re changing is the project name. In actual usage just run dsco init and answer the prompts.
[6]:
!echo "foo\n\n\n\n\n\n" | dsco init
Using template: /Users/ap186098/.pyenv/versions/3.7.5/lib/python3.7/site-packages/dsco/project_template
Creating project in /private/tmp/dsco_overview
project_name [MyProject]: author [Discovery]: version [0.1]: year [2020]: project_port [8001]: Select python_version:
1 - 3.7
2 - 3.6
Choose from 1, 2 [1]:
[7]:
cd foo
/private/tmp/dsco_overview/foo
[8]:
!ls
README.md definitions.py flask_config.ini landing_page
__pycache__ docker-compose.yml flask_run.py pyproject.toml
builder docs foo tests
One of the nice things about using dsco to create your project is the consistency in project structure. Let’s point out a few of the key features.
A standard project README file.
The builder directory contains everything necessary to build the project containers including docker files and ansible playbooks.
[9]:
cat definitions.py
#!/usr/bin/env python
# * encoding: utf-8
import os
def _test_dir(dir_path):
if not os.path.isdir(dir_path):
raise IOError(f"Expected specs directory at {dir_path}")
ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
SQL_DIR = os.path.join(ROOT_DIR, "foo", "sql")
_test_dir(SQL_DIR)
This is a placeholder for defining a few useful project variables. If you need access to the project root directory you can import it from definitions.
This defines the service definitions: dev, prod, and debug for use with docker-compose. This is where they are tied to the appropriate docker file.
[10]:
!tree docs
docs
├── assets
│ ├── banner.png
│ ├── space-shuttle-launch.png
│ └── sphinxheader.png
├── documentation
│ ├── html
│ │ └── plotly.js
│ └── source
│ ├── Makefile
│ ├── __pycache__
│ │ └── conf.cpython-37.pyc
│ ├── _static
│ │ └── plotly.js
│ ├── conf.py
│ └── index.rst
├── index.html
└── reports
├── html
│ └── plotly.js
└── source
├── Makefile
├── __init__.py
├── __pycache__
│ ├── __init__.cpython-37.pyc
│ └── conf.cpython-37.pyc
├── _static
│ └── plotly.js
├── components.png
├── conf.py
├── index.rst
├── reports_template.ipynb
└── resources.ipynb
11 directories, 21 files
The docs directory is important. Dsco distinguishes between two different types of docs: documentation and reports.
Documentation is project documentation. The how and why of your code.
Reports is meant to be a place to put your exploratory analysis.
Dsco facilitates the production of a static html representation of each of these types of documentation. Put any .rst or notebook files in the source directory and generate html with either dsco documentation -g or dsco reports -g.
[11]:
cat flask_config.ini
[uwsgi]
socket = 127.0.0.1:3031
mount = /webapp=flask_run.py
callable=app
master = true
processes = 2
chmod-socket = 660
vacuum = true
die-on-term = true
manage-script-name = true
The container is running Flask with uwsgi. This is the configuration file. Don’t mess with it unless you know what you’re doing.
[12]:
cat flask_run.py
"""App entry point."""
from foo import create_app
app = create_app()
if __name__ == "__main__":
app.run(host="0.0.0.0", debug=True)
This is the flask entry point. Again, don’t mess with it unless you know what you’re doing.
[13]:
!tree foo
foo
├── __init__.py
├── __pycache__
│ ├── __init__.cpython-37.pyc
│ ├── connection.cpython-37.pyc
│ ├── flask_main.cpython-37.pyc
│ ├── foo.cpython-37.pyc
│ ├── helpers.cpython-37.pyc
│ ├── queries.cpython-37.pyc
│ └── routes.cpython-37.pyc
├── connection.py
├── dashapp1
│ ├── __init__.py
│ └── __pycache__
│ └── __init__.cpython-37.pyc
├── flask_main.py
├── foo.py
├── helpers.py
├── queries.py
├── routes.py
└── sql
├── all_test_queries.sql
└── events.sql
4 directories, 18 files
This is your python package directory. It will have whatever name you provided in the --init command for [project_name]. This is a standard setup for creating python packages.
If you’re not creating a package, don’t worry about it. It’s not taking up a lot of overhead. Just create notebooks in the docs/reports/source directory and leave this alone.
If you are creating a package, there are some nice features in this package directory you should know about.
Opinionated perspective
In the introduction, we said we take an opinionated perspective on the data science development perspective. That started with the first step of doing EDA in jupyter notebooks.
After EDA, the next level of maturity is to build some sort of dashboard. Dsco sets up the scaffolding for using plotly | Dash in conjunction with Flask to build your dashboard.
The flask app is created in flask_main.py. Flask routes are configured in routes.py. This is where you can create rest API routes that would return data.
The dash app is created in dashapp1/__init__.py.
nginx is configured to handle requests to /. Nginx also handles serving the static content generated in docs. This can be found at /reports/ and /documentation/.
flask is configured to handle requests to /webapp/.
dash is configured to handle requests to /webapp/dash.
The landing_page directory contains the html for the page you see when you connect to the container at http://<container>:<port>/.
These files manage our python dependencies. For more information see http://https://python-poetry.org/.
TODO: At the moment this is more of a place holder. The intention is to build scaffolding around testing and wrap that in something like dsco test.
Now that we have our project directory, it’s time to build our development image. We do that with dsco build.
[14]:
!dsco build --help
usage: dsco build [-h] [--dev] [--prod] [--debug] [--all]
optional arguments:
-h, --help show this help message and exit
--dev build dev image
--prod build prod image
--debug build debug image
--all build all images
[15]:
# this produces a lot of output from ansible and takes ~8 minutes
!dsco build
docker-compose build dev
Building dev
Step 1/8 : FROM python:3.7-slim as base
---> 69afd9568c9d
Step 2/8 : COPY . /srv
---> ca403a3be728
Step 3/8 : SHELL ["/bin/bash", "-c"]
---> Running in a233c691f4c8
Removing intermediate container a233c691f4c8
---> 218704534db2
Step 4/8 : RUN pip install ansible toml
---> Running in 46ddbe2dbe06
Collecting ansible
Downloading ansible-2.9.6.tar.gz (14.2 MB)
Collecting toml
Downloading toml-0.10.0-py2.py3-none-any.whl (25 kB)
Collecting jinja2
Downloading Jinja2-2.11.1-py2.py3-none-any.whl (126 kB)
Collecting PyYAML
Downloading PyYAML-5.3.1.tar.gz (269 kB)
Collecting cryptography
Downloading cryptography-2.8-cp34-abi3-manylinux2010_x86_64.whl (2.3 MB)
Collecting MarkupSafe>=0.23
Downloading MarkupSafe-1.1.1-cp37-cp37m-manylinux1_x86_64.whl (27 kB)
Collecting six>=1.4.1
Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting cffi!=1.11.3,>=1.8
Downloading cffi-1.14.0-cp37-cp37m-manylinux1_x86_64.whl (400 kB)
Collecting pycparser
Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: ansible, PyYAML
Building wheel for ansible (setup.py): started
Building wheel for ansible (setup.py): finished with status 'done'
Created wheel for ansible: filename=ansible-2.9.6-py3-none-any.whl size=16164182 sha256=b918f090a8f388bb08738ab81fdc49613a99e8023ad25111c2bf505809a73299
Stored in directory: /root/.cache/pip/wheels/59/31/d9/a309a1fce4216dd912b705ce76035bff3ed5bbd990853bad97
Building wheel for PyYAML (setup.py): started
Building wheel for PyYAML (setup.py): finished with status 'done'
Created wheel for PyYAML: filename=PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl size=44619 sha256=857ca518426df24e3866d57274075d09b8ff7aa886860205f32154d98e799d8c
Stored in directory: /root/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653
Successfully built ansible PyYAML
Installing collected packages: MarkupSafe, jinja2, PyYAML, six, pycparser, cffi, cryptography, ansible, toml
Successfully installed MarkupSafe-1.1.1 PyYAML-5.3.1 ansible-2.9.6 cffi-1.14.0 cryptography-2.8 jinja2-2.11.1 pycparser-2.20 six-1.14.0 toml-0.10.0
Removing intermediate container 46ddbe2dbe06
---> 080ee5a933be
Step 5/8 : FROM base as dev
---> 080ee5a933be
Step 6/8 : SHELL ["/bin/bash", "-c"]
---> Running in f385ccba1915
Removing intermediate container f385ccba1915
---> 0d0997f87dd0
Step 7/8 : RUN cd /srv/builder/ansible && ansible-playbook build_dev_container.yml
---> Running in 9cc14a874aa5
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
PLAY [localhost] ***************************************************************
TASK [python3-apt : Get location of python site-packages] **********************
changed: [localhost]
TASK [python3-apt : Download python3-apt] **************************************
changed: [localhost]
TASK [python3-apt : get name of file we downloaded] ****************************
ok: [localhost]
TASK [python3-apt : extract package] *******************************************
changed: [localhost]
TASK [python3-apt : rename .so files] ******************************************
changed: [localhost]
TASK [python3-apt : cleanup downloads - python3-apt*.deb] **********************
changed: [localhost]
TASK [python3-apt : cleanup downloads - /tmp/python3-apt] **********************
changed: [localhost]
TASK [os-prod : Install os production dependencies] ****************************
changed: [localhost] => (item=gcc)
changed: [localhost] => (item=python3-dev)
[WARNING]: Updating cache and auto-installing missing dependency: python3-apt
TASK [os-prod : basic shell settings] ******************************************
changed: [localhost]
TASK [os-dev : Install os development dependencies] ****************************
changed: [localhost] => (item=make)
changed: [localhost] => (item=wget)
changed: [localhost] => (item=curl)
changed: [localhost] => (item=git)
changed: [localhost] => (item=vim)
changed: [localhost] => (item=pandoc)
TASK [poetry : check if poetry is installed] ***********************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "$HOME/.poetry/bin/poetry --version", "delta": "0:00:00.003655", "end": "2020-03-27 15:45:59.181337", "msg": "non-zero return code", "rc": 127, "start": "2020-03-27 15:45:59.177682", "stderr": "/bin/sh: 1: /root/.poetry/bin/poetry: not found", "stderr_lines": ["/bin/sh: 1: /root/.poetry/bin/poetry: not found"], "stdout": "", "stdout_lines": []}
...ignoring
TASK [poetry : download] *******************************************************
changed: [localhost]
TASK [poetry : install and configure] ******************************************
changed: [localhost]
TASK [poetry : remove download] ************************************************
changed: [localhost]
TASK [poetry : modify .bashrc for poetry] **************************************
changed: [localhost]
TASK [python-dev : remove poetry.lock to install dependencies from scratch] ****
skipping: [localhost]
TASK [python-dev : create python development environment] **********************
changed: [localhost]
TASK [supervisor : install supervisor] *****************************************
changed: [localhost]
TASK [supervisor : config supervisor] ******************************************
changed: [localhost]
TASK [supervisor : remove supervisor.sock] *************************************
ok: [localhost]
TASK [supervisor : start] ******************************************************
changed: [localhost]
TASK [uwsgi : check that flask_config.ini exists] ******************************
ok: [localhost]
TASK [uwsgi : fail if file does not exist] *************************************
skipping: [localhost]
TASK [uwsgi : install uwsgi] ***************************************************
changed: [localhost]
TASK [uwsgi : setup config file for supervisor] ********************************
changed: [localhost]
TASK [uwsgi : Create the log directory] ****************************************
changed: [localhost]
TASK [uwsgi : update supervisor] ***********************************************
changed: [localhost]
TASK [jupyter-config : Check for jupyter] **************************************
changed: [localhost]
TASK [jupyter-config : Check if jupyter_extensions (var) installed] ************
changed: [localhost] => (item=jupyter-contrib-nbextensions)
changed: [localhost] => (item=qgrid)
TASK [jupyter-config : install if missing] *************************************
skipping: [localhost] => (item=['jupyter-contrib-nbextensions', {'cmd': '$HOME/.poetry/bin/poetry show jupyter-contrib-nbextensions', 'stdout': 'Skipping virtualenv creation, as specified in config file.\nname : jupyter-contrib-nbextensions\nversion : 0.5.1\ndescription : A collection of Jupyter nbextensions.\n\ndependencies\n - ipython-genutils *\n - jupyter-contrib-core >=0.3.3\n - jupyter-core *\n - jupyter-highlight-selected-word >=0.1.1\n - jupyter-latex-envs >=1.3.8\n - jupyter-nbextensions-configurator >=0.4.0\n - lxml *\n - nbconvert >=4.2\n - notebook >=4.0\n - pyyaml *\n - tornado *\n - traitlets >=4.1', 'stderr': '', 'rc': 0, 'start': '2020-03-27 15:50:46.201458', 'end': '2020-03-27 15:50:47.602127', 'delta': '0:00:01.400669', 'changed': True, 'invocation': {'module_args': {'_raw_params': '$HOME/.poetry/bin/poetry show jupyter-contrib-nbextensions', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Skipping virtualenv creation, as specified in config file.', 'name : jupyter-contrib-nbextensions', 'version : 0.5.1', 'description : A collection of Jupyter nbextensions.', '', 'dependencies', ' - ipython-genutils *', ' - jupyter-contrib-core >=0.3.3', ' - jupyter-core *', ' - jupyter-highlight-selected-word >=0.1.1', ' - jupyter-latex-envs >=1.3.8', ' - jupyter-nbextensions-configurator >=0.4.0', ' - lxml *', ' - nbconvert >=4.2', ' - notebook >=4.0', ' - pyyaml *', ' - tornado *', ' - traitlets >=4.1'], 'stderr_lines': [], 'failed': False, 'item': 'jupyter-contrib-nbextensions', 'ansible_loop_var': 'item'}])
skipping: [localhost] => (item=['qgrid', {'cmd': '$HOME/.poetry/bin/poetry show qgrid', 'stdout': 'Skipping virtualenv creation, as specified in config file.\nname : qgrid\nversion : 1.3.0\ndescription : An Interactive Grid for Sorting and Filtering DataFrames in\n Jupyter Notebook\n\ndependencies\n - ipywidgets >=7.0.0\n - notebook >=4.0.0\n - pandas >=0.18.0', 'stderr': '', 'rc': 0, 'start': '2020-03-27 15:50:47.790825', 'end': '2020-03-27 15:50:49.200780', 'delta': '0:00:01.409955', 'changed': True, 'invocation': {'module_args': {'_raw_params': '$HOME/.poetry/bin/poetry show qgrid', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Skipping virtualenv creation, as specified in config file.', 'name : qgrid', 'version : 1.3.0', 'description : An Interactive Grid for Sorting and Filtering DataFrames in', ' Jupyter Notebook', '', 'dependencies', ' - ipywidgets >=7.0.0', ' - notebook >=4.0.0', ' - pandas >=0.18.0'], 'stderr_lines': [], 'failed': False, 'item': 'qgrid', 'ansible_loop_var': 'item'}])
TASK [jupyter-config : create jupyter log dir] *********************************
changed: [localhost]
TASK [jupyter-config : .jupyter directory] *************************************
changed: [localhost]
TASK [jupyter-config : jupyter_notebook_config.py] *****************************
changed: [localhost]
TASK [jupyter-config : nbconfig directory] *************************************
changed: [localhost]
TASK [jupyter-config : notebook.json] ******************************************
changed: [localhost]
TASK [jupyter-config : custom directory] ***************************************
changed: [localhost]
TASK [jupyter-config : custom.js] **********************************************
changed: [localhost]
TASK [jupyter-config : install nbextensions] ***********************************
changed: [localhost]
TASK [jupyter-config : copy jupyter.conf file to supervisor dir] ***************
changed: [localhost]
TASK [jupyter-config : update supervisor] **************************************
changed: [localhost]
TASK [sphinx-config : Check if sphinx is installed] ****************************
changed: [localhost]
TASK [sphinx-config : Install os requirements] *********************************
ok: [localhost] => (item=make)
ok: [localhost] => (item=pandoc)
TASK [sphinx-config : check if sphinx_extensions (var) are installed] **********
changed: [localhost] => (item=nbsphinx)
changed: [localhost] => (item=guzzle-sphinx-theme)
TASK [sphinx-config : install if missing] **************************************
skipping: [localhost] => (item=['nbsphinx', {'cmd': '. $HOME/.poetry/env\npoetry show nbsphinx\n', 'stdout': 'Skipping virtualenv creation, as specified in config file.\nname : nbsphinx\nversion : 0.4.3\ndescription : Jupyter Notebook Tools for Sphinx\n\ndependencies\n - docutils *\n - jinja2 *\n - nbconvert !=5.4\n - nbformat *\n - sphinx >=1.6.3\n - traitlets *', 'stderr': '', 'rc': 0, 'start': '2020-03-27 15:51:03.336287', 'end': '2020-03-27 15:51:04.717443', 'delta': '0:00:01.381156', 'changed': True, 'invocation': {'module_args': {'_raw_params': '. $HOME/.poetry/env\npoetry show nbsphinx\n', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Skipping virtualenv creation, as specified in config file.', 'name : nbsphinx', 'version : 0.4.3', 'description : Jupyter Notebook Tools for Sphinx', '', 'dependencies', ' - docutils *', ' - jinja2 *', ' - nbconvert !=5.4', ' - nbformat *', ' - sphinx >=1.6.3', ' - traitlets *'], 'stderr_lines': [], 'failed': False, 'item': 'nbsphinx', 'ansible_loop_var': 'item'}])
skipping: [localhost] => (item=['guzzle-sphinx-theme', {'cmd': '. $HOME/.poetry/env\npoetry show guzzle-sphinx-theme\n', 'stdout': 'Skipping virtualenv creation, as specified in config file.\nname : guzzle-sphinx-theme\nversion : 0.7.11\ndescription : Sphinx theme used by Guzzle.\n\ndependencies\n - Sphinx >=1.2b1', 'stderr': '', 'rc': 0, 'start': '2020-03-27 15:51:04.903681', 'end': '2020-03-27 15:51:06.271434', 'delta': '0:00:01.367753', 'changed': True, 'invocation': {'module_args': {'_raw_params': '. $HOME/.poetry/env\npoetry show guzzle-sphinx-theme\n', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Skipping virtualenv creation, as specified in config file.', 'name : guzzle-sphinx-theme', 'version : 0.7.11', 'description : Sphinx theme used by Guzzle.', '', 'dependencies', ' - Sphinx >=1.2b1'], 'stderr_lines': [], 'failed': False, 'item': 'guzzle-sphinx-theme', 'ansible_loop_var': 'item'}])
TASK [sphinx-config : Get location of python site-packages] ********************
changed: [localhost]
TASK [sphinx-config : modify guzzle theme (make it wider)] *********************
changed: [localhost]
TASK [sphinx-config : build documentation] *************************************
changed: [localhost]
TASK [nginx : install nginx] ***************************************************
changed: [localhost]
TASK [nginx : add nginx site-available default] ********************************
changed: [localhost]
TASK [nginx : link sites-enabled to sites-available] ***************************
ok: [localhost]
TASK [nginx : restart nginx] ***************************************************
changed: [localhost]
PLAY RECAP *********************************************************************
localhost : ok=47 changed=42 unreachable=0 failed=0 skipped=4 rescued=0 ignored=1
Removing intermediate container 9cc14a874aa5
---> 10b203751251
Step 8/8 : LABEL com.teradata.dsco=True
---> Running in d7b58242fb18
Removing intermediate container d7b58242fb18
---> 881efba8cbb3
Successfully built 881efba8cbb3
Successfully tagged foo_dev:latest
========================================================================================
build execution time: 485.67613315582275 seconds
========================================================================================
Generally this build takes about 8 minutes (see execution time at the end).
We can see that it built if we run dsco ls. We should see foo_dev:latest listed.
[16]:
# grep is just filtering the content. Generally just use 'dsco ls'
!dsco ls | egrep 'localhost|foo|NAME'
localhost
NAME LINK ID STATUS SIZE
foo_dev:latest 881efba8cbb3 1 second ago 1.75GB
The next step is to create a container from the image. If the image is already built, this is really fast, if the image has not been built, this will first build the image, and then create the container, which will take about 8 minutes.
[17]:
!dsco up
docker-compose up -d dev
Creating foo_dev_1 ...
ting foo_dev_1 ... done
========================================================================================
up execution time: 1.4652199745178223 seconds
========================================================================================
We can use dsco ls to see if the container is up.
[18]:
# grep is just filtering the content. Generally just use 'dsco ls'
!dsco ls | egrep 'localhost|foo|NAME'
localhost
NAME LINK ID STATUS SIZE
foo_dev:latest 881efba8cbb3 5 seconds ago 1.75GB
foo_dev_1 http://localhost:8001 591858ba06da Up Less than a second
We can access the container at the specified link. If the container is not running you will see this.
[19]:
# we can use docker commands to interact with our containers as well
# this one stops the container without deleting it
!docker stop foo_dev_1
foo_dev_1
[20]:
# grep is just filtering the content. Generally just use 'dsco ls'
!dsco ls | egrep 'localhost|foo|NAME'
localhost
NAME LINK ID STATUS SIZE
foo_dev:latest 881efba8cbb3 6 seconds ago 1.75GB
foo_dev_1 http://_________:8001 591858ba06da Exited (143) Less than a second ago
[21]:
# start it again ...
!dsco up
docker-compose up -d dev
Starting foo_dev_1 ...
ting foo_dev_1 ... done
========================================================================================
up execution time: 1.2733056545257568 seconds
========================================================================================
[22]:
!dsco go --help
usage: dsco go [-h] [--dev] [--prod] [--debug] [--all]
optional arguments:
-h, --help show this help message and exit
--dev dev container
--prod prod container
--debug debug container
--all all containers
Here we have used dsco build and dsco up to build and launch our dev container. We also could have used dsco go.
dsco go runs dsco up and then launches a web browser connected to the container. If a container is already running, the only effect will be to open a web browser.
[23]:
!dsco go
docker-compose up -d dev
foo_dev_1 is up-to-date
========================================================================================
up execution time: 0.8994390964508057 seconds
========================================================================================
========================================================================================
go execution time: 1.089184045791626 seconds
========================================================================================
If we try to go to access our reports or documentation through the links on the landing page we will be met with an error 403 message. This is because we have not built it yet.
[24]:
# build documentation
!dsco documentation --generate
docker exec -it $(docker ps --all --filter 'label=com.docker.compose.project=foo' --filter 'label=com.docker.compose.service=dev' --format {{.Names}}) bash -c 'cd /srv/docs/documentation/source && make html'
Running Sphinx v2.4.4
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
/usr/local/lib/python3.7/site-packages/guzzle_sphinx_theme/guzzle_sphinx_theme/layout.html:200: RemovedInSphinx30Warning: To modify script_files in the theme is deprecated. Please insert a <script> tag directly in your theme instead.
generating indices... genindexdone
writing additional pages... searchdone
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.
The HTML pages are in ../html.
[25]:
# build reports
!dsco reports --generate
docker exec -it $(docker ps --all --filter 'label=com.docker.compose.project=foo' --filter 'label=com.docker.compose.service=dev' --format {{.Names}}) bash -c 'cd /srv/docs/reports/source && make html'
Running Sphinx v2.4.4
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 3 source files that are out of date
updating environment: [new config] 3 added, 0 changed, 0 removed
reading sources... [100%] resources
/srv/docs/reports/source/reports_template.ipynb:24: WARNING: File not found: '/notebook/notebooks/reports/reports_template.ipynb'
/srv/docs/reports/source/reports_template.ipynb:24: WARNING: File not found: '/reports/reports_template.html#'
/srv/docs/reports/source/reports_template.ipynb:41: WARNING: File not found: '/notebook/edit/reports/index.rst'
looking for now-outdated files... none found
pickling environment... done
checking consistency... /srv/docs/reports/source/resources.ipynb: WARNING: document isn't included in any toctree
done
preparing documents... done
/usr/local/lib/python3.7/site-packages/guzzle_sphinx_theme/guzzle_sphinx_theme/layout.html:200: RemovedInSphinx30Warning: To modify script_files in the theme is deprecated. Please insert a <script> tag directly in your theme instead.
/usr/local/lib/python3.7/site-packages/guzzle_sphinx_theme/guzzle_sphinx_theme/layout.html:200: RemovedInSphinx30Warning: To modify script_files in the theme is deprecated. Please insert a <script> tag directly in your theme instead.
writing output... [100%] resources
generating indices... genindexdone
writing additional pages... searchdone
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 4 warnings.
The HTML pages are in ../html.
Now we should be able to access the starting points of our documentation and reports.
Of course, there’s just some basic scaffolding in place. To create the documentation write some content in reStructuredText files or jupyter notebooks and add these files to index.rst in the source/ directory.
As a simple example:
[26]:
%%writefile docs/reports/source/example.rst
Example
=======
This is an example .rst file
Writing docs/reports/source/example.rst
[27]:
%%writefile docs/reports/source/index.rst
.. Template documentation master file, created by
sphinx-quickstart
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to foo's documentation!
===============================
.. toctree::
:maxdepth: 2
:caption: Contents:
reports_template
example
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
Overwriting docs/reports/source/index.rst
[28]:
!dsco reports --generate
docker exec -it $(docker ps --all --filter 'label=com.docker.compose.project=foo' --filter 'label=com.docker.compose.service=dev' --format {{.Names}}) bash -c 'cd /srv/docs/reports/source && make html'
Running Sphinx v2.4.4
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 1 added, 1 changed, 0 removed
reading sources... [100%] index
looking for now-outdated files... none found
pickling environment... done
checking consistency... /srv/docs/reports/source/resources.ipynb: WARNING: document isn't included in any toctree
done
preparing documents... done
/usr/local/lib/python3.7/site-packages/guzzle_sphinx_theme/guzzle_sphinx_theme/layout.html:200: RemovedInSphinx30Warning: To modify script_files in the theme is deprecated. Please insert a <script> tag directly in your theme instead.
writing output... [100%] index
generating indices... genindexdone
writing additional pages... searchdone
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 1 warning.
The HTML pages are in ../html.
The rendering isn’t perfect here, but you can see a link to Example under the Welcome to foo's documentation! section.
[29]:
display(HTML("docs/reports/html/index.html"))
And another imperfect rendering of the Example page we created.
[30]:
display(HTML("docs/reports/html/example.html"))
[31]:
!dsco ls --help
usage: dsco ls [-h] [-r]
optional arguments:
-h, --help show this help message and exit
-r, --remote list remote containers using docker over ssh
We’ve already been using dsco ls to see the containers and images that are present on our local machine.
We can also use dsco ls -r to see containers we have running on remote machines. This leverages docker’s ability to run docker commands over ssh by setting the DOCKER_HOST environment variable.
For this to work, you need to have keyless ssh set up to your remote host, and then create a $HOME/.dsco/settings.json file that contains the following:
{
"remote": [
{
"name": "my_name_for_my_machine",
"properties": {
"ip": "fully_qualified_domain_name"
},
"env": {
"DOCKER_HOST": "ssh://user@fqdn"
}
},
{
"name": "my_name_for_my_machine2",
"properties": {
"ip": "fully_qualified_domain_name"
},
"env": {
"DOCKER_HOST": "ssh://user@fqdn"
}
}
]
}
[32]:
# grep is just filtering the content. Generally just use 'dsco ls'
!dsco ls | egrep 'localhost|foo|NAME'
localhost
NAME LINK ID STATUS SIZE
foo_dev:latest 881efba8cbb3 25 seconds ago 1.75GB
foo_dev_1 http://localhost:8001 591858ba06da Up 16 seconds
[33]:
!dsco shell --help
usage: dsco shell [-h] [--dev] [--prod] [--debug]
optional arguments:
-h, --help show this help message and exit
--dev attach to dev
--prod attach to prod
--debug attach to debug
Use the dsco shell command to open a bash shell inside your container.
This is the preferred way to work with your project dependencies.
[34]:
!dsco code --help
usage: dsco code [-h] [-r REMOTE]
optional arguments:
-h, --help show this help message and exit
-r REMOTE, --remote REMOTE
connect to a container with docker-machine
Use dsco code to launch a session of Microsoft’s vscode IDE. After you install the Remote Development Extension, you can use the Remote Explorer to attach to running containers. Open up the /srv directory and enjoy a “local-like” development experience.
Even more amazing is the ability to attach to containers running on remote machines. If you have remote machines set up as described in the --ls section, you can simple type dsco code -r my_name_for_my_machine. When vscode opens you can use Remote Explorer to attach to the listed containers, but this time you will see the containers running on the remote machine. Super Cool!!!
You can read more about using vscode with containers here: https://code.visualstudio.com/docs/remote/containers.
[35]:
!dsco restart --help
usage: dsco restart [-h] [--flask] [--nginx]
optional arguments:
-h, --help show this help message and exit
--flask restart flask
--nginx restart nginx
When you’re working on creating your Dash dashboard, you will need to restart flask to see the changes you made. dsco restart is an easy way to do this. Without options it will restart both nginx and flask. Use the flags to be more specific.
[36]:
!dsco restart
docker exec -it $(docker ps --all --filter 'label=com.docker.compose.project=foo' --filter 'label=com.docker.compose.service=dev' --format {{.Names}}) bash -c "service nginx restart"
[....] Restarting nginx: nginx7[ ok 8.
docker exec -it $(docker ps --all --filter 'label=com.docker.compose.project=foo' --filter 'label=com.docker.compose.service=dev' --format {{.Names}}) bash -c "supervisorctl restart uwsgi && supervisorctl status"
uwsgi: stopped
uwsgi: started
jupyter RUNNING pid 56, uptime 0:00:21
uwsgi RUNNING pid 446, uptime 0:00:01
[37]:
!dsco update_port --help
usage: dsco update_port [-h] [--dev DEV] [--prod PROD] [--debug DEBUG]
[base_port]
positional arguments:
base_port update ports dev: base_port, prod: base_port+1000, debug:
base_port-500
optional arguments:
-h, --help show this help message and exit
--dev DEV update dev port, overwrites base_port
--prod PROD update prod port, overwrites base_port
--debug DEBUG update debug port, overwrites base_port
When you set up your project, you were prompted for a port number. By default, this is the port of the dev container. The prod container is assigned port + 1000 and the debug container is assigned port - 500.
If you’re running multiple containers on the same machine, they all need to be on different ports. To change the port you can use dsco update_port
If you enter dsco update_port 8001: you will get the default behavior described above. If you want to override this behavior, simply add the appropriate flag and specify the desired port.
For example:
dsco update_port 8001 --prod 8501
dev = 8001
prod = 8501
debug = 7501
dsco update_port --debug 7600
dev = unchanged
prod = unchanged
debug = 7600
Once you’ve changed the port you will need to run dsco up make the changes take effect.
Note the port is 8001.
[38]:
# grep is just filtering the content. Generally just use 'dsco ls'
!dsco ls | egrep 'localhost|foo|NAME'
localhost
NAME LINK ID STATUS SIZE
foo_dev:latest 881efba8cbb3 32 seconds ago 1.75GB
foo_dev_1 http://localhost:8001 591858ba06da Up 24 seconds
[39]:
!dsco update_port 8023
Still 8001, we need to restart it.
[40]:
# grep is just filtering the content. Generally just use 'dsco ls'
!dsco ls | egrep 'localhost|foo|NAME'
localhost
NAME LINK ID STATUS SIZE
foo_dev:latest 881efba8cbb3 34 seconds ago 1.75GB
foo_dev_1 http://localhost:8001 591858ba06da Up 25 seconds
[41]:
!dsco up
docker-compose up -d dev
Recreating foo_dev_1 ...
eating foo_dev_1 ... done
========================================================================================
up execution time: 1.6713709831237793 seconds
========================================================================================
Now we see it has updated.
[42]:
# grep is just filtering the content. Generally just use 'dsco ls'
!dsco ls | egrep 'localhost|foo|NAME'
localhost
NAME LINK ID STATUS SIZE
foo_dev:latest 881efba8cbb3 37 seconds ago 1.75GB
foo_dev_1 http://localhost:8023 6ccb824c7cf9 Up Less than a second
Put it back.
[43]:
!dsco update_port 8001
[44]:
!dsco up
docker-compose up -d dev
Recreating foo_dev_1 ...
eating foo_dev_1 ... done
========================================================================================
up execution time: 1.6443562507629395 seconds
========================================================================================
[45]:
# grep is just filtering the content. Generally just use 'dsco ls'
!dsco ls | egrep 'localhost|foo|NAME'
localhost
NAME LINK ID STATUS SIZE
foo_dev:latest 881efba8cbb3 40 seconds ago 1.75GB
foo_dev_1 http://localhost:8001 e266fa9944c7 Up Less than a second
[46]:
!dsco rm --help
usage: dsco rm [-h] [--dev] [--prod] [--debug] [--all]
optional arguments:
-h, --help show this help message and exit
--dev remove dev
--prod remove prod
--debug remove debug
--all remove dev, prod, and debug
dsco rm will by default remove the dev container.[47]:
!dsco rm
docker rm -f $(docker ps --all --filter 'label=com.docker.compose.project=foo' --filter 'label=com.docker.compose.service=dev' --format {{.Names}})
foo_dev_1
[48]:
# grep is just filtering the content. Generally just use 'dsco ls'
!dsco ls | egrep 'localhost|foo|NAME'
localhost
NAME LINK ID STATUS SIZE
foo_dev:latest 881efba8cbb3 42 seconds ago 1.75GB
[49]:
!dsco rmi --help
usage: dsco rmi [-h] [--dev] [--prod] [--debug] [--all] [--force]
optional arguments:
-h, --help show this help message and exit
--dev remove dev
--prod remove prod
--debug remove debug
--all remove dev, prod, and debug
--force remove corresponding containers
dsco rmi will by default remove the dev image.dsco rmi will fail is there is a container running that is based on the image you are trying to delete.
[50]:
!dsco up
# grep is just filtering the content. Generally just use 'dsco ls'
!dsco ls | egrep 'localhost|foo|NAME'
docker-compose up -d dev
Creating foo_dev_1 ...
ting foo_dev_1 ... done
========================================================================================
up execution time: 1.297410011291504 seconds
========================================================================================
localhost
NAME LINK ID STATUS SIZE
foo_dev:latest 881efba8cbb3 45 seconds ago 1.75GB
foo_dev_1 http://localhost:8001 71cb8ff45a4e Up Less than a second
[51]:
!dsco rmi
docker rmi foo_dev
Error response from daemon: conflict: unable to remove repository reference "foo_dev" (must force) - container 71cb8ff45a4e is using its referenced image 881efba8cbb3
You can force the removal of both with the --force flag.
[52]:
!dsco rmi --all --force
docker rm -f $(docker ps --all --filter 'label=com.docker.compose.project=foo' --filter 'label=com.docker.compose.service=dev' --format {{.Names}})
foo_dev_1
docker rm -f $(docker ps --all --filter 'label=com.docker.compose.project=foo' --filter 'label=com.docker.compose.service=prod' --format {{.Names}})
"docker rm" requires at least 1 argument.
See 'docker rm --help'.
Usage: docker rm [OPTIONS] CONTAINER [CONTAINER...]
Remove one or more containers
docker rm -f $(docker ps --all --filter 'label=com.docker.compose.project=foo' --filter 'label=com.docker.compose.service=debug' --format {{.Names}})
"docker rm" requires at least 1 argument.
See 'docker rm --help'.
Usage: docker rm [OPTIONS] CONTAINER [CONTAINER...]
Remove one or more containers
docker rmi --force foo_dev
Untagged: foo_dev:latest
Deleted: sha256:881efba8cbb309ac440aa19d1897bac2d45da338a2058263659deae2fdb79969
Deleted: sha256:10b203751251eba5827454031a24170918d487a4abd0a064ee887881295cdcd6
Deleted: sha256:5ebffafa2085339cb1362fd74950805f059d9b8ac210db84799417d43210583d
Deleted: sha256:0d0997f87dd086a98ae5b65111b80e21101a209f846f13be2f35c9e7e5ba1342
Deleted: sha256:080ee5a933be553f3f0f13528e1a14cd87d94900564f03106bc841189ef937b9
Deleted: sha256:54a2868080c30299e6afdc583b6f912346787839fd3ef8264fd636536cc6bc4a
Deleted: sha256:218704534db2f9964bc3183fb8044afa45a17a5d2c60e1eac770d66bbbc91fc4
Deleted: sha256:ca403a3be7289870c52f39d6eb2694c1a98fe9821b653e5f04d39f015c39b413
Deleted: sha256:0b57602ea2d3ef35d86704ffac55602d7c3af7d0b2fcf75fe2f430bf3605dcd9
docker rmi --force foo_prod
Error: No such image: foo_prod
docker rmi --force foo_debug
Error: No such image: foo_debug
[53]:
# grep is just filtering the content. Generally just use 'dsco ls'
!dsco ls | egrep 'localhost|foo|NAME'
localhost
NAME LINK ID STATUS SIZE
Finish by deleting our project directory.
[54]:
cd /tmp
/private/tmp
[55]:
rm -rf /tmp/dsco_overview
[56]:
ls
4E65C18E-676A-4F76-A157-7B16F54B6B1F/ com.google.Keystone/
65B7725E-4EA0-4B57-A357-C39E564D676B/ com.trendmicro.tmsm.run.plist
8EFCF5EB-0232-4BA2-BE29-E882B44305EF/ cvcd/
AlTest1.err ipcroboform8.3.6
AlTest1.out port.61100=
B531E494-1D8B-44CA-960F-D5D1DDE836CF/ port.61201=
ExmanProcessMutex port.61301=
ICRC_ROLL.lck port.61302=
MSS_Menulet port.61304=
adobegc.log port.61401=
com.adobe.AdobeIPCBroker.ctrl-ap186098= port.61501=
com.apple.launchd.TAzDf6vxI5/ powerlog/
com.apple.launchd.wiVigXgrId/ tmux-1937012048/
com.apple.launchd.wyf1x4DwRU/ wbxtrace/
[ ]: