Project Jupyter and IPython¶

• 100% of confirmed issues should have a milestone.
• An issue should never be closed without a milestone.
• All pull requests should have a milestone.
• All issues closed should be marked with the next release milestone, next backport milestone, or no action.
• Open issues should only lack a milestone if:
  • more clarification is required (label: needs-info)
• In general, there will be four milestones with open issues:
  • next minor release. This milestone contains issues that should be backported for the next minor release. See below for information on backporting.
  • next major release. This should be the default milestone of all issues. As the release approaches, issues can be explicitly bumped to next release + 1.
  • next major release + 1. Only issues that we are confident will not be included in the next release go here. This milestone should be mostly empty until relatively close to release.
  • wishlist. This is the milestone for issues that we have no immediate plans to address.
• The remaining milestone is no action for open or closed issues that require no action:
  • Not actually an issue (e.g. questions, discussion, etc.)
  • Duplicate of an existing Issue
  • Closed because we won’t fix it
  • When an issue is closed with no action, it means that the issue will not be fixed, or it was not an issue at all.
• When closing an issue, it should always have one of these milestones:
  • next minor release because the issue has been addressed
  • next major release because the issue has been addressed
  • no action because the issue will not be addressed, or it is a duplicate/non-issue.

In general: When in doubt, mark with next release. We can always push back when we get closer to a given release.

Issues should always be labeled once they are confirmed (not necessary for issues that are still being clarified, or may be duplicates or not issues at all).

Some significant labels:

• needs-info: issue needs more information from submitter before progress can be made

• bug: errors are raised, or unintended behavior occurs

• enhancement: improvements that are not bugs

• backport-X.Y.Z: Any fix for this issue should be backported to the maintenance branch. backports are expressed with milestones, starting with 2.1.

• prio-foo: a priority level for ranking issues - nonessential, but prio-high/low are useful for explicitly promoting/demoting issues, particularly when nearing release.

• ClosedPR: This issue is a reminder for a PR that was closed for going stale.

• sprint-friendly: Obvious or easy fixes, where

All confirmed issues should at least have a bug or enhancement label, and be marked with any affected components (e.g parallel, qtconsole, htmlnotebook), or particular sources of error (e.g. py3k or unicode) if we have appropriate labels.

The sprint-friendly label is probably the best place for new contributors to start.

• All work is submitted via Pull Requests.
• Pull Requests can be submitted as soon as there is code worth discussing. Pull Requests track the branch, so you can continue to work after the PR is submitted. Review and discussion can begin well before the work is complete, and the more discussion the better. The worst case is that the PR is closed.
• Pull Requests that have stalled should be closed (see [[our policy on closing PRs|Dev: Closing Pull Requests]])
• Pull Requests should always be made against master (backporting PRs is described below).
• Pull Requests should be tested, if feasible:
  • bugfixes should include regression tests
  • new behavior should at least get minimal exercise

Travis does a pretty good job testing IPython and Pull Requests, but it may make sense to manually perform tests (possibly with our test_pr script), particularly for PRs that affect IPython.parallel or Windows.

When opening a new issue, please take the following steps:

1. Search GitHub and/or Google for your issue to avoid duplicate reports. Keyword searches for your error messages are most helpful.

2. If possible, try updating to master and reproducing your issue, because we may have already fixed it.

3. Try to include a minimal reproducible test case

4. Include relevant system information. Start with the output of:

   python -c "import IPython; print(IPython.sys_info())"
   

And include any relevant package versions, depending on the issue, such as matplotlib, numpy, Qt, Qt bindings (PyQt/PySide), tornado, web browser, etc.

• We should keep an A.x maintenance branch for backporting fixes from master.
• That branch shall be called A.x, e.g. 2.x, not 2.1. This way, there is only one maintenance branch per release series.
• When an Issue is determined to be appropriate for backporting, it should be marked with the A.B milestone.
• Any Pull Request which addresses a backport issue should also receive the same milestone.
• Patches are backported to the maintenance branch by applying the pull request patch to the maintenance branch (currently with the backport_pr script).

The code does what it’s supposed to!

IPython has to work on:

• Linux of various kinds, Windows & Mac
• Python 2 & 3

Much of our code base deals with strings and unicode. This needs to be done in a manner that is unicode aware and works on Python 2 and 3. [This article] (http://www.joelonsoftware.com/articles/Unicode.html) is a good intro to unicode.

Coding style refers to how source code is formatted and how variables, functions, methods and classes are named. Your code should follow our coding style, which is described [[here|Dev: Coding style]].

The code should be well organized, and have inline comments where appropriate. When we look at the code, it should be clear what it’s doing and why. It should not break abstractions that we have established in the project.

If it fixes a bug, the pull request should ideally add an automated test that fails without the fix, and passes with it. Normally it should be sufficient to copy an existing test and tweak it. New functionality should come with its own tests as well. Details about testing IPython can be found [[here|Dev: Testing]].

Don’t forget to update docstrings, and any relevant parts of the official documentation. New features or significant changes warrant an entry in the What’s New section too. Details about documenting IPython can be found [[here|Dev: Documenting IPython]].

In general, we follow the standard Python style conventions as described in Python’s PEP 8, the official Python Style Guide.

Other general comments:

• In a large file, top level classes and functions should be separated by 2 lines to make it easier to separate them visually.
• Use 4 spaces for indentation, never use hard tabs.
• Keep the ordering of methods the same in classes that have the same methods. This is particularly true for classes that implement similar interfaces and for interfaces that are similar.

For naming conventions, we also follow the guidelines of PEP 8. Some of the existing code doesn’t honor this perfectly, but for all new and refactored IPython code, we’ll use:

• All lowercase module names. Long module names can have words separated by underscores (really_long_module_name.py), but this is not required. Try to use the convention of nearby files.
• CamelCase for class names.
• lowercase_with_underscores for methods, functions, variables and attributes.
• Implementation-specific private methods will use _single_underscore_prefix. Names with a leading double underscore will only be used in special cases, as they makes subclassing difficult (such names are not easily seen by child classes).
• Occasionally some run-in lowercase names are used, but mostly for very short names or where we are implementing methods very similar to existing ones in a base class (like runlines() where runsource() and runcode() had established precedent).
• The old IPython codebase has a big mix of classes and modules prefixed with an explicit IP of ip. This is not necessary and all new code should not use this prefix. The only case where this approach is justified is for classes or functions which are expected to be imported into external namespaces and a very generic name (like Shell) that is likely to clash with something else. However, if a prefix seems absolutely necessary the more specific IPY or ipy are preferred.
• All JavaScript code should follow these naming conventions as well.

In general, objects should declare, in their class, all attributes the object is meant to hold throughout its life. While Python allows you to add an attribute to an instance at any point in time, this makes the code harder to read and requires methods to constantly use checks with hasattr() or try/except calls. By declaring all attributes of the object in the class header, there is a single place one can refer to for understanding the object’s data interface, where comments can explain the role of each variable and when possible, sensible deafaults can be assigned.

If an attribute is meant to contain a mutable object, it should be set to None in the class and its mutable value should be set in the object’s constructor. Since class attributes are shared by all instances, failure to do this can lead to difficult to track bugs. But you should still set it in the class declaration so the interface specification is complete and documented in one place.

A simple example:

class Foo(object):
    # X does..., sensible default given:
    x = 1
    # y does..., default will be set by constructor
    y = None
    # z starts as an empty list, must be set in constructor
    z = None

    def __init__(self, y):
        self.y = y
        self.z = []

When starting a new Python file for IPython, you can use the following template as a starting point that has a few common things pre-written for you.

All standalone documentation should be written in plain text (.txt) files using reStructuredText [reStructuredText]_ for markup and formatting. All such documentation should be placed in the directory docs/source of the IPython source tree. Or, when appropriate, a suitably named subdirectory should be used. The documentation in this location will serve as the main source for IPython documentation.

The actual HTML and PDF docs are built using the Sphinx [Sphinx]_ documentation generation tool. Once you have Sphinx installed, you can build the html docs yourself by doing:

$ cd ipython-mybranch/docs
$ make html

Our usage of Sphinx follows that of matplotlib [Matplotlib]_ closely. We are using a number of Sphinx tools and extensions written by the matplotlib team and will mostly follow their conventions, which are nicely spelled out in their documentation guide [MatplotlibDocGuide]_. What follows is thus a abridged version of the matplotlib documentation guide, taken with permission from the matplotlib team.

If you are reading this in a web browser, you can click on the “Show Source” link to see the original reStricturedText for the following examples.

A bit of Python code:

for i in range(10):
    print i,
print "A big number:",2**34

An interactive Python session:

>>> from IPython.utils.path import get_ipython_dir
>>> get_ipython_dir()
'/home/fperez/.config/ipython'

An IPython session:

In [7]: import IPython

In [8]: print "This IPython is version:",IPython.__version__
This IPython is version: 0.9.1

In [9]: 2+4
Out[9]: 6

A bit of shell code:

cd /tmp
echo "My home directory is: $HOME"
ls

Good docstrings are very important. Unfortunately, Python itself only provides a rather loose standard for docstrings [PEP257]_, and there is no universally accepted convention for all the different parts of a complete docstring. However, the NumPy project has established a very reasonable standard, and has developed some tools to support the smooth inclusion of such docstrings in Sphinx-generated manuals. Rather than inventing yet another pseudo-standard, IPython will be henceforth documented using the NumPy conventions; we carry copies of some of the NumPy support tools to remain self-contained, but share back upstream with NumPy any improvements or fixes we may make to the tools.

The NumPy documentation guidelines [NumPyDocGuide]_ contain detailed information on this standard, and for a quick overview, the NumPy example docstring [NumPyExampleDocstring]_ is a useful read.

For user-facing APIs, we try to be fairly strict about following the above standards (even though they mean more verbose and detailed docstrings). Wherever you can reasonably expect people to do introspection with:

In [1]: some_function?

the docstring should follow the NumPy style and be fairly detailed.

For purely internal methods that are only likely to be read by others extending IPython itself we are a bit more relaxed, especially for small/short methods and functions whose intent is reasonably obvious. We still expect docstrings to be written, but they can be simpler. For very short functions with a single-line docstring you can use something like:

def add(a, b):
   """The sum of two numbers.
   """
   code

and for longer multiline strings:

def add(a, b):
   """The sum of two numbers.

   Here is the rest of the docs.
   """
   code

Here are two additional PEPs of interest regarding documentation of code. While both of these were rejected, the ideas therein form much of the basis of docutils (the machinery to process reStructuredText):

• Docstring Processing System Framework

• Docutils Design Specification

  note

  In the past IPython used epydoc so currently many docstrings still use epydoc conventions. We will update them as we go, but all new code should be documented using the NumPy standard.

The built docs are stored in a separate repository. Through some github magic, they’re automatically exposed as a website. It works like this:

• You will need to have sphinx and latex installed. In Ubuntu, install texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended. Install the latest version of sphinx from PyPI (pip install sphinx).
• Ensure that the development version of IPython is the first in your system path. You can either use a virtualenv, or modify your PYTHONPATH.
• Switch into the docs directory, and run make gh-pages. This will build your updated docs as html and pdf, then automatically check out the latest version of the docs repository, copy the built docs into it, and commit your changes.
• Open the built docs in a web browser, and check that they’re as expected.
• (When building the docs for a new tagged release, you will have to add its link to index.rst, then run python build_index.py to update index.html. Commit the change.)
• Upload the docs with git push. This only works if you have write access to the docs repository.
• If you are building a version that is not the current dev branch, nor a tagged release, then you must run gh-pages.py directly with python gh-pages.py <version>, and not with make gh-pages.

We are trying to keep things simple and with a minimum of new moving parts:

• Meetings happen on Tuesdays at 10am California time (i.e. UTC-8 or -7 depending on the time of year).
• We broadcast the meeting as it happens via G+ and leave the public YouTube link afterwards.
• During the meeting, all chat that needs to happen by typing can be done on our Gitter chat room.
• We keep a running document with minutes of the meeting on HackPad where we summarize main points. (2015 part 1)

We welcome and encourage help from others in updating these minutes during the meeting, but we’ll make no major effort in ensuring that they are a detailed and accurate record of the whole discussion. We simply don’t have the time for that.

You can find a list of the videos on the ipythondev YouTube user page.

It is extremely important that all code contributed to IPython has tests. Tests should be written as unittests, doctests or other entities that the IPython test system can detect. See below for more details on this.

Each subpackage in IPython should have its own tests directory that contains all of the tests for that subpackage. All of the files in the tests directory should have the word “tests” in them to enable the testing framework to find them.

In docstrings, examples (either using IPython prompts like In [1]: or ‘classic’ python >>> ones) can and should be included. The testing system will detect them as doctests and will run them; it offers control to skip parts or all of a specific doctest if the example is meant to be informative but shows non-reproducible information (like filesystem data).

If a subpackage has any dependencies beyond the Python standard library, the tests for that subpackage should be skipped if the dependencies are not found. This is very important so users don’t get tests failing simply because they don’t have dependencies.

The testing system we use is an extension of the nose test runner. In particular we’ve developed a nose plugin that allows us to paste verbatim IPython sessions and test them as doctests, which is extremely important for us.

You can run IPython from the source download directory without even installing it system-wide or having configure anything, by typing at the terminal:

python2 -c "import IPython; IPython.start_ipython();"

To start the webbased notebook you can use:

python2 -c "import IPython; IPython.start_ipython(['notebook']);"

In order to run the test suite, you must at least be able to import IPython, even if you haven’t fully installed the user-facing scripts yet (common in a development environment). You can then run the tests with:

python -c "import IPython; IPython.test()"

Once you have installed IPython either via a full install or using:

python setup.py develop

you will have available a system-wide script called iptest that runs the full test suite. You can then run the suite with:

iptest  [args]

By default, this excludes the relatively slow tests for IPython.parallel. To run these, use iptest --all.

Please note that the iptest tool will run tests against the code imported by the Python interpreter. If the command python setup.py symlink has been previously run then this will always be the test code in the local directory via a symlink. However, if this command has not been run for the version of Python being tested, there is the possibility that iptest will run the tests against an installed version of IPython.

Regardless of how you run things, you should eventually see something like:

**********************************************************************
Test suite completed for system with the following information:
{'commit_hash': '144fdae',
 'commit_source': 'repository',
 'ipython_path': '/home/fperez/usr/lib/python2.6/site-packages/IPython',
 'ipython_version': '0.11.dev',
 'os_name': 'posix',
 'platform': 'Linux-2.6.35-22-generic-i686-with-Ubuntu-10.10-maverick',
 'sys_executable': '/usr/bin/python',
 'sys_platform': 'linux2',
 'sys_version': '2.6.6 (r266:84292, Sep 15 2010, 15:52:39) \n[GCC 4.4.5]'}

Tools and libraries available at test time:
   curses matplotlib pymongo qt sqlite3 tornado wx wx.aui zmq

Ran 9 test groups in 67.213s

Status:
OK

If not, there will be a message indicating which test group failed and how to rerun that group individually. For example, this tests the IPython.utils subpackage, the -v option shows progress indicators:

$ iptest IPython.utils -- -v
..........................SS..SSS............................S.S...
.........................................................
----------------------------------------------------------------------
Ran 125 tests in 0.119s

OK (SKIP=7)

Because the IPython test machinery is based on nose, you can use all nose syntax. Options after -- are passed to nose. For example, this lets you run the specific test test_rehashx inside the test_magic module:

$ iptest IPython.core.tests.test_magic:test_rehashx -- -vv
IPython.core.tests.test_magic.test_rehashx(True,) ... ok
IPython.core.tests.test_magic.test_rehashx(True,) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.100s

OK

When developing, the --pdb and --pdb-failures of nose are particularly useful, these drop you into an interactive pdb session at the point of the error or failure respectively: iptest mymodule -- --pdb.

The system information summary printed above is accessible from the top level package. If you encounter a problem with IPython, it’s useful to include this information when reporting on the mailing list; use:

.. code:: python

from IPython import sys_info print sys_info()

and include the resulting information in your query.

We have a script that fetches a pull request from Github, merges it with master, and runs the test suite on different versions of Python. This uses a separate copy of the repository, so you can keep working on the code while it runs. To run it:

python tools/test_pr.py -p 1234

The number is the pull request number from Github; the -p flag makes it post the results to a comment on the pull request. Any further arguments are passed to iptest.

This requires the requests and keyring packages.

By now IPython has a reasonable test suite, so the best way to see what’s available is to look at the tests directory in most subpackages. But here are a few pointers to make the process easier.

def ranfunc():
"""A function with some random output.

   Normal examples are verified as usual:
   >>> 1+3
   4

   But if you put '# random' in the output, it is ignored:
   >>> 1+3
   junk goes here...  # random

   >>> 1+2
   again,  anything goes #random
   if multiline, the random mark is only needed once.

   >>> 1+2
   You can also put the random marker at the end:
   # random

   >>> 1+2
   # random
   .. or at the beginning.

   More correct input is properly verified:
   >>> ranfunc()
   'ranfunc'
"""
return 'ranfunc'

def random_all():
"""A function where we ignore the output of ALL examples.

Examples:

  # all-random

  This mark tells the testing machinery that all subsequent examples
  should be treated as random (ignoring their output).  They are still
  executed, so if a they raise an error, it will be detected as such,
  but their output is completely ignored.

  >>> 1+3
  junk goes here...

  >>> 1+3
  klasdfj;

In [8]: print 'hello'
world  # random

In [9]: iprand()
Out[9]: 'iprand'
"""
return 'iprand'

# The docstring for full_path doctests differently on win32 (different path
# separator) so just skip the doctest there, and use a null decorator
# elsewhere:

doctest_deco = dec.skip_doctest if sys.platform == 'win32' else dec.null_deco

@doctest_deco
def full_path(startPath,files):
    """Make full paths for all the listed files, based on startPath..."""

    # function body follows...

def doctest_time():
"""
In [10]: %time None
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s
"""

cd IPython/html/tests/casperjs;
casperjs test --includes=util.js test_cases

casperjs test --includes=util.js --port=8889 test_cases

casperjs test --includes=util.js  test_cases/execute_code_cell.js

casper.notebook_test(function () {
    this.evaluate(function () {
        var cell = IPython.notebook.get_cell(0);
        cell.set_text('import time\nfor x in range(3):\n    time.sleep(1)');
        cell.execute();
    });


    // interrupt using menu item (Kernel -> Interrupt)
    this.thenClick('li#int_kernel');

    this.wait_for_output(0);

    this.then(function () {
        var result = this.get_output_cell(0);
        this.test.assertEquals(result.ename, 'KeyboardInterrupt', 'keyboard interrupt (mouseclick)');
    });

    // run cell 0 again, now interrupting using keyboard shortcut
    this.thenEvaluate(function () {
        cell.clear_output();
        cell.execute();
    });

    // interrupt using Ctrl-M I keyboard shortcut
    this.thenEvaluate( function() {
        IPython.utils.press_ghetto(IPython.utils.keycodes.I)
    });

    this.wait_for_output(0);

    this.then(function () {
        var result = this.get_output_cell(0);
        this.test.assertEquals(result.ename, 'KeyboardInterrupt', 'keyboard interrupt (shortcut)');
    });
});

This section is a set of notes on the key points of the IPython testing needs, that were used when writing the system and should be kept for reference as it eveolves.

Testing IPython in full requires modifications to the default behavior of nose and doctest, because the IPython prompt is not recognized to determine Python input, and because IPython admits user input that is not valid Python (things like %magics and !system commands.

We basically need to be able to test the following types of code:

• 1. Pure Python files containing normal tests. These are not a problem, since Nose will pick them up as long as they conform to the (flexible) conventions used by nose to recognize tests.
• 2. Python files containing doctests. Here, we have two possibilities:
• The prompts are the usual >>> and the input is pure Python.
• The prompts are of the form In [1]: and the input can contain extended IPython expressions.

In the first case, Nose will recognize the doctests as long as it is called with the --with-doctest flag. But the second case will likely require modifications or the writing of a new doctest plugin for Nose that is IPython-aware.

• 3. ReStructuredText files that contain code blocks. For this type of file, we have three distinct possibilities for the code blocks:
• They use >>> prompts.
• They use In [1]: prompts.
• They are standalone blocks of pure Python code without any prompts.

The first two cases are similar to the situation #2 above, except that in this case the doctests must be extracted from input code blocks using docutils instead of from the Python docstrings.

In the third case, we must have a convention for distinguishing code blocks that are meant for execution from others that may be snippets of shell code or other examples not meant to be run. One possibility is to assume that all indented code blocks are meant for execution, but to have a special docutils directive for input that should not be executed.

For those code blocks that we will execute, the convention used will simply be that they get called and are considered successful if they run to completion without raising errors. This is similar to what Nose does for standalone test functions, and by putting asserts or other forms of exception-raising statements it becomes possible to have literate examples that double as lightweight tests.

• 4. Extension modules with doctests in function and method docstrings. Currently Nose simply can’t find these docstrings correctly, because the underlying doctest DocTestFinder object fails there. Similarly to #2 above, the docstrings could have either pure python or IPython prompts.

Of these, only 3-c (reST with standalone code blocks) is not implemented at this point.

You can set some env variables to note previous release tag and current release milestone, version, and git tag:

PREV_RELEASE=rel-1.0.0
MILESTONE=1.1
VERSION=1.1.0
TAG="rel-$VERSION"
BRANCH=master

These will be used later if you want to copy/paste, or you can just type the appropriate command when the time comes. These variables are not used by scripts (hence no export).

• If a major release:

• merge any pull request notes into what’s new:

  python tools/update_whatsnew.py
  

• update docs/source/whatsnew/development.rst, to ensure it covers the major points.

• move the contents of development.rst to versionX.rst

• generate summary of GitHub contributions, which can be done with:

  python tools/github_stats.py --milestone $MILESTONE > stats.rst
  

which may need some manual cleanup. Add the cleaned up result and add it to docs/source/whatsnew/github-stats-X.rst (make a new file, or add it to the top, depending on whether it is a major release). You can use:

git log --format="%aN <%aE>" $PREV_RELEASE... | sort -u -f

to find duplicates and update .mailmap. Before generating the GitHub stats, verify that all closed issues and pull requests have appropriate milestones. This search should return no results.

This does all the file checking and building that the real release script will do. This will let you do test installations, check that the build procedure runs OK, etc. You may want to also do a test build of the docs.

Edit IPython/core/release.py to have the current version.

Commit the changes to release.py and jsversion:

git commit -am "release $VERSION"
git push origin $BRANCH

Create and push the tag:

git tag -am "release $VERSION" "$TAG"
git push origin --tags

Update release.py back to x.y-dev or x.y-maint, and push:

git commit -am "back to development"
git push origin $BRANCH

cd tools && ./release

This makes the tarballs, zipfiles, and wheels. It posts them to archive.ipython.org and registers the release with PyPI.

This will require that you have current wheel, Python 3.4 and Python 2.7.

• release announcement (news, announcements)
• update current version and download links
• (If major release) update links on the documentation page

This should include i) highlights and ii) a link to the html version of the What’s new section of the documentation.

Post to mailing list, and link from Twitter.

• close the milestone you just released
• open new milestone for (x, y+1), if it doesn’t exist already

Pure python code is supported by the optional argument python. In this pure python syntax you do not include the output from the python interpreter. The following markup:

.. code:: python

   foo = 'bar'
   print foo
   foo = 2
   foo**2

Renders as

foo = 'bar'
print foo
foo = 2
foo**2

We can even plot from python, using the savefig decorator, as well as, suppress output with a semicolon

@savefig plot_simple_python.png width=4in
plot([1,2,3]);

Similarly, std err is inserted

foo = 'bar'
foo[)

Comments are handled and state is preserved

# comments are handled
print foo

If you don’t see the next code block then the options work.

ioff()
ion()

Multi-line input is handled.

line = 'Multi\
        line &\
        support &\
        works'
print line.split('&')

Functions definitions are correctly parsed

def square(x):
    """
    An overcomplicated square function as an example.
    """
    if x < 0:
        x = abs(x)
    y = x * x
    return y

And persist across sessions

print square(3)
print square(-2)

Pretty much anything you can do with the ipython code, you can do with with a simple python script. Obviously, though it doesn’t make sense to use the doctest option.

• decode, encode: Shortcuts to decode or encode strings, using sys.stdin.encoding by default, and using replacement characters on errors.
• str_to_unicode, unicode_to_str, str_to_bytes, bytes_to_str: Convert to/from the platform’s standard str type (bytes in Python 2, unicode in Python 3). Each function is a no-op on one of the two platforms.
• cast_unicode, cast_bytes: Accept unknown unicode or byte strings, and convert them accordingly.
• cast_bytes_py2: Casts unicode to byte strings on Python 2, but doesn’t do anything on Python 3.

• input: Refers to raw_input on Python 2, input on Python 3 (needed because 2to3 only converts calls to raw_input, not assignments to other names).
• builtin_mod_name: The string name you import to get the builtins (__builtin__ –> builtins).
• isidentifier: Checks if a string is a valid Python identifier.
• open: Simple wrapper for Python 3 unicode-enabled open. Similar to codecs.open, but allows universal newlines. The current implementation only supports the very simplest use.
• MethodType: types.MethodType from Python 3. Takes only two arguments: function, instance. The class argument for Python 2 is filled automatically.
• doctest_refactor_print: Can be called on a string or a function (or used as a decorator). In Python 3, it converts print statements in doctests to print() calls. 2to3 does this for real doctests, but we need it in several other places. It simply uses a regex, which is good enough for the current cases.
• u_format: Where tests use the repr() of a unicode string, it should be written '{u}"thestring"', and fed to this function, which will produce 'u"thestring"' for Python 2, and '"thestring"' for Python 3. Can also be used as a decorator, to work on a docstring.
• execfile: Makes a return on Python 3 (where it’s no longer a builtin), and upgraded to handle Unicode filenames on Python 2.

Miscellaneous

Notebook contents API.

Kernel API

Sessions API

Clusters API

This chart shows the web-services in the single directory IPython notebook.

This chart shows the architecture for the IPython notebook website.

This chart shows the Web services that act on the kernels and clusters.

• command_mode.Cell
• create.Cell
• delete.Cell
• edit_mode.Cell
• select.Cell
• output_appended.OutputArea

• preset_activated.CellToolbar
• preset_added.CellToolbar

• app_initialized.DashboardApp
• sessions_loaded.Dashboard

• execution_request.Kernel
• input_reply.Kernel
• kernel_autorestarting.Kernel
• kernel_busy.Kernel
• kernel_connected.Kernel
• kernel_connection_failed.Kernel
• kernel_created.Kernel
• kernel_created.Session
• kernel_dead.Kernel
• kernel_dead.Session
• kernel_disconnected.Kernel
• kernel_idle.Kernel
• kernel_interrupting.Kernel
• kernel_killed.Kernel
• kernel_killed.Session
• kernel_ready.Kernel
• kernel_reconnecting.Kernel
• kernel_restarting.Kernel
• kernel_starting.Kernel
• send_input_reply.Kernel
• shell_reply.Kernel
• spec_changed.Kernel

import os
os._exit(1)

import os
from IPython.kernel.connect import get_connection_file
with open(get_connection_file(), 'w') as f:
    f.write("garbage")
os._exit(1)

• app_initialized.NotebookApp
• autosave_disabled.Notebook
• autosave_enabled.Notebook
• checkpoint_created.Notebook
• checkpoint_delete_failed.Notebook
• checkpoint_deleted.Notebook
• checkpoint_failed.Notebook
• checkpoint_restore_failed.Notebook
• checkpoint_restored.Notebook
• checkpoints_listed.Notebook
• command_mode.Notebook
• edit_mode.Notebook
• list_checkpoints_failed.Notebook
• notebook_load_failed.Notebook
• notebook_loaded.Notebook
• notebook_loading.Notebook
• notebook_rename_failed.Notebook
• notebook_renamed.Notebook
• notebook_restoring.Notebook
• notebook_save_failed.Notebook
• notebook_saved.Notebook
• notebook_saving.Notebook
• rename_notebook.Notebook
• selected_cell_type_changed.Notebook
• set_dirty.Notebook
• set_next_input.Notebook
• trust_changed.Notebook

• open_with_text.Pager
• rebuild.QuickHelp

Install boot2docker. There are multiple ways to install, depending on your environment. See the boot2docker docs.

$ brew install boot2docker docker

$ boot2docker init

$ boot2docker up

$ $(boot2docker shellinit)

$ boot2docker ip
192.168.59.103

Use the Dockerfile in the cloned ipython directory to build a Docker image.

$ cd ipython
$ docker build --rm -t ipython .

Run a container using the new image. We mount the entire ipython source tree on the host into the container at /srv/ipython to enable changes we make to the source on the host immediately reflected in the container.

# change to the root of the git clone
$ cd ipython
$ docker run -it --rm -p 8888:8888 --workdir /srv/ipython --name ipython-dev -v `pwd`:/srv/ipython ipython /bin/bash

To list the running container from another shell on the host:

$ $(boot2docker shellinit)
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                    NAMES
f6065f206519        ipython             "/bin/bash"         1 minutes ago       Up 1 minutes        0.0.0.0:8888->8888/tcp   ipython-dev

Once in the container, you’ll need to uninstall the ipython package and re-install in editable mode to enable your dev changes to be reflected in your environment.

container $ pip uninstall ipython

# pip install ipython in editable mode
container $ cd /srv
container $ ls
ipython
container $ pip install -e ipython

On your host, run the following command to get the IP of the boot2docker VM if you forgot:

# on host
$ boot2docker ip
192.168.59.103

Then visit it in your browser:

# browser
http://192.168.59.103:8888

As a shortcut on a Mac, you can run the following in a terminal window (or make it a bash alias):

$ open http://$(boot2docker ip 2>/dev/null):8888

The kerneltest tool is part of IPython.testing and is also included in the scripts similar to iptest. This takes 2 parameters - the name of the kernel to test and the test script file. The test script file should be in json format as described in the next section.

kerneltest python test_script.json

You can also pass in an optional message spec version to the command. At the moment only the version 5 is supported, but as newer versions are released this can be used to test the kernel against a specific version of the kernel.

kerneltest python test_script.json 5

The kernel to be tested needs to be installed and the kernelspec available in the user IPython directory. The tool will instantiate the kernel and send the commands over ZMQ. For each command executed on the kernel, the tool will validate the reply to ensure that it matches the message specification. In some cases the output is also checked, but the reply is always returned and printed out on the console. This can be used to validate that apart from meeting the message spec the kernel also produced the correct output.

The test script file is a simple json file that specifies the command to execute and the test code to execute for the command.

{
    "command":{
           "test_code":<code>
    }
}

For some commands in the message specification like kernel_info there is no need to specify the test_code parameter. The tool validates if it has all the inputs needed to execute the command and will print out an error to the console if it finds a missing parameter. Since the validation is built in, and only required parameters are passed, it is possible to add additional fields in the json file for test documentation.

{
    "command":{
           "test_name":"sample test",
           "test_description":"sample test to show how the test script file is created",
           "test_code":<code>
    }
}

A sample test script for the redis kernel will look like this

{
  "execute":{
    "test_code":"get a",
    "comments":"test basic code execution"
  },
  "complete":{
    "test_code":"get",
    "comments":"test getting command auto complete"
  },
  "kernel_info":{
    "comments":"simple kernel info check"
  },
  "single_payload":{
    "test_code":"get a",
    "comments":"test one payload"
  },
  "history_tail":{
    "test_code":"get a",
    "comments":"test tail history"
  },
  "history_range":{
    "test_code":"get a",
    "comments":"test range history"
  },
  "history_search":{
    "test_code":"get a",
    "comments":"test search history"
  }
}

• docs directory : All source files for documentation go here.
• readthedocs.yml : configuration file for readthedocs to build using conda

• source directory : contains all content source files in .rst, .md, or .ipynb
• makefile : used by Sphinx to build the docs
• environment.yml : conda build instructions

• conf.py : Sphinx configuration file
• index.rst of contents.rst : Sphinx master table of contents file
• _static directory : contains images, drawings, icons
• _templates directory: overrides theme templates and layouts
• build directory : html files generated by Sphinx (do not check this directory into GitHub)

• [ ] Add a link to documentation in repo description (requires GitHub repo privileges)
• [ ] Add badges to README (Edit README.md and submit pull request)
• [ ] Add resources section to README (Edit README.md and submit pull request)

Dated: 1-4-2016 Revised: 1-7-2016

Webhooks and services can be enabled in GitHub repo settings to allow third party services such as ReadTheDocs. The ReadTheDocs service rebuilds the project documentation whenever a pull request is merged into the GitHub repo.