SaltyCrane Blog — Notes on JavaScript and web development

Do you have a lot of short, single-use, private functions in your Python code?

Do you have a lot of short, single-use, private functions in your Python code? For example, below is some stubbed out authentication code I've been working on. It checks if a user's password is correct and updates the hash algorithm to use bcrypt. The 4 private functions with the leading underscore are from 1 to 10 lines long and are only used by the check_password function. These functions are part of a larger module with about 20 functions. I don't like that these 4 functions add clutter to the module and are not grouped with the function that uses them, check_password.

def _get_password_hash_from_db(email_address):
    """Get the user's password hash from the database.
    """


def _determine_password_hash_algorithm(password_hash):
    """Determine the hash algorithm.
    """


def _hash_password_old(password):
    """This is the OLD password hash algorithm.
    """


def _hash_existing_password_bcrypt(password, db_password_hash):
    """This is the NEW algorithm used for hashing existing passwords.
    """


def check_password(email_address, password):
    """Check if a user's supplied password is correct.
    """
    db_password_hash = _get_password_hash_from_db(email_address)
    hash_alg = _determine_password_hash_algorithm(db_password_hash)
    if hash_alg == 'BCRYPT':
        input_password_hash = _hash_existing_password_bcrypt(password, db_password_hash)
    else:
        input_password_hash = _hash_password_old(password)
    password_correct = (input_password_hash == db_password_hash)
    if password_correct and hash_alg != 'BCRYPT':
        call_change_password(email_address, password)
    return password_correct


def call_change_password(email_address, new_password):
    """Change the user's password.
    """

Sometimes, in cases like this, I move the 4 private functions to be nested functions inside check_password. I like how the functions are grouped together and that the module is not littered with extraneous functions. However, the inner functions are not easily testable and I don't see many people doing this.

def check_password(email_address, password):
    """Check if a user's supplied password is correct.
    """

    def get_password_hash_from_db(email_address):
        """Get the user's password hash from the database.
        """

    def determine_password_hash_algorithm(password_hash):
        """Determine the hash algorithm.
        """

    def hash_password_old(password):
        """This is the OLD password hash algorithm.
        """

    def hash_existing_password_bcrypt(password, db_password_hash):
        """This is the NEW algorithm used for hashing existing passwords.
        """

    db_password_hash = get_password_hash_from_db(email_address)
    hash_alg = determine_password_hash_algorithm(db_password_hash)
    if hash_alg == 'BCRYPT':
        input_password_hash = hash_existing_password_bcrypt(password, db_password_hash)
    else:
        input_password_hash = hash_password_old(password)
    password_correct = (input_password_hash == db_password_hash)
    if password_correct and hash_alg != 'BCRYPT':
        call_change_password(email_address, password)
    return password_correct


def call_change_password(email_address, new_password):
    """Change the user's password.
    """

Another option is to create a PasswordChecker class instead. This seems the most powerful and now the private methods are testable. However, this adds more overhead and I hear Jack Diederich telling me to Stop Writing Classes!

class _PasswordChecker(object):
    """Check if a user's supplied password is correct.
    """

    @staticmethod
    def _get_password_hash_from_db(email_address):
        """Get the user's password hash from the database.
        """

    @staticmethod
    def _determine_password_hash_algorithm(password_hash):
        """Determine the hash algorithm.
        """

    @staticmethod
    def _hash_password_old(password):
        """This is the OLD password hash algorithm.
        """

    @staticmethod
    def _hash_existing_password_bcrypt(password, db_password_hash):
        """This is the NEW algorithm used for hashing existing passwords.
        """

    def __call__(self, email_address, password):
        db_password_hash = self._get_password_hash_from_db(email_address)
        hash_alg = self._determine_password_hash_algorithm(db_password_hash)
        if hash_alg == 'BCRYPT':
            input_password_hash = self._hash_existing_password_bcrypt(password, db_password_hash)
        else:
            input_password_hash = self._hash_password_old(password)
        password_correct = (input_password_hash == db_password_hash)
        if password_correct and hash_alg != 'BCRYPT':
            call_change_password(email_address, password)
        return password_correct


check_password = _PasswordChecker()


def call_change_password(email_address, new_password):
    """Change the user's password.
    """

Maybe the solution is to break up the module into smaller modules which act like the class above? However this might leave me with some unevenly sized modules. How do you handle this?

How to expose a Flask local development server to the public using SSH remote port forwarding

Here is how to run a Flask local development server on your local machine and expose it to the public via a remote server you have control over. This uses SSH remote port forwarding which is a converse of local port forwarding described here: How to run a Django local development server on a remote machine and access it in your browser on your local machine using SSH port forwarding

  1. On the remote host, edit the sshd_config file (mine was located at /etc/ssh/sshd_config) to allow remote hosts to connect to ports forwarded for the client:
    GatewayPorts yes
  2. On the remote host, restart the SSH server:
    $ sudo service sshd restart 
    
  3. On the local host, SSH to the remote host:
    $ ssh -v -R 50051:localhost:5000 [email protected] 
    
  4. On the local host, run the Flask dev server:
    $ python runserver.py localhost 5000 
    
  5. Go to http://my.remotehost.com:50051 in the browser

Using RemoteForward in your ~/.ssh/config

You can also achieve the same results by using the RemoteForward in your ~/.ssh/config file:

Host myremote
  User eliot
  HostName my.remotehost.com
  RemoteForward 50051 localhost:5000

References

See also

localtunnel by Jeff Lindsay exposes your local development server without requiring a public remote server.

When is the try-finally block used in Python?

The finally block is used to define clean-up actions. Why is the finally block needed? Why can't the clean up actions be put after the try/except/else block? This works in some cases, but if there is a return, break, or continue, or an unhandled exception inside the try, except, or else clauses, that code will never be executed. The finally block executes even in these conditions.

try:
    print 'Inside try'
    raise Exception
finally:
    print 'Inside finally'
print 'Never get here'

Results:

Inside try
Inside finally
Traceback (most recent call last):
  File "tmp.py", line 13, in 
    raise Exception
Exception

Reference: http://docs.python.org/2/tutorial/errors.html#defining-clean-up-actions

Using Python's gzip and StringIO to compress data in memory

I needed to gzip some data in memory that would eventually end up saved to disk as a .gz file. I thought, That's easy, just use Python's built in gzip module.

However, I needed to pass the data to pycurl as a file-like object. I didn't want to write the data to disk and then read it again just to pass to pycurl. I thought, That's easy also-- just use Python's cStringIO module.

The solution did end up being simple, but figuring out the solution was a lot harder than I thought. Below is my roundabout process of finding the simple solution.

Here is my setup/test code. I am running Python 2.7.3 on Ubuntu 12.04.

import cStringIO
import gzip


STUFF_TO_GZIP = """Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?"""
FILENAME = 'myfile.json.gz'


def pycurl_simulator(fileobj):

    # Get the file size
    fileobj.seek(0, 2)
    filesize = fileobj.tell()
    fileobj.seek(0, 0)

    # Read the file data
    fout = open(FILENAME, 'wb')
    fout.write(fileobj.read())
    fout.close()

    return filesize

Try 1: seek from the end fails

Here is my first attempt using cStringIO with the gzip module.

def try1_seek_from_end_fails():

    ftemp = cStringIO.StringIO()
    fgzipped = gzip.GzipFile(
        filename=FILENAME, mode='wb', fileobj=ftemp)
    fgzipped.write(STUFF_TO_GZIP)
    filesize = pycurl_simulator(fgzipped)
    print filesize

I got this exception:

        Traceback (most recent call last):
          File "tmp.py", line 232, in <module>
            try1_seek_from_end_fails()
          File "tmp.py", line 83, in try1_seek_from_end_fails
            filesize = pycurl_simulator(fgzipped)
          File "tmp.py", line 25, in pycurl_simulator
            fileobj.seek(0, 2)
          File "/usr/lib/python2.7/gzip.py", line 415, in seek
            raise ValueError('Seek from end not supported')
        ValueError: Seek from end not supported

It turns out the gzip object doesn't support seeking from the end. See this thread on the Python mailing list: http://mail.python.org/pipermail/python-list/2009-January/519398.html

Try 2: data is not compressed

What if we don't seek() from the end and just tell() where we are? (It should be at the end after doing a write(), right?) Unfortunately, this gave me the uncompressed size.

Reading from the GzipFile object also gave me an error saying that I couldn't read from a writable object.

def try2_data_is_not_compressed():

    ftemp = cStringIO.StringIO()
    fgzipped = gzip.GzipFile(
        filename=FILENAME, mode='wb', fileobj=ftemp)
    fgzipped.write(STUFF_TO_GZIP)
    filesize = fgzipped.tell()
    print filesize

Try 5: file much too small

I googled, then looked at the source code for gzip.py. I found that the compressed data was in the StringIO object. So I performed my file operations on it instead of the GzipFile object. Now I was able to write the data out to a file. However, the size of the file was much too small.

def try5_file_much_too_small():

    fgz = cStringIO.StringIO()
    gzip_obj = gzip.GzipFile(
        filename=FILENAME, mode='wb', fileobj=fgz)
    gzip_obj.write(STUFF_TO_GZIP)
    filesize = pycurl_simulator(fgz)
    print filesize

Try 6: unexpected end of file

I saw there was a flush() method in the source code. I added a call to flush(). This time, I got a reasonable file size, however, when trying to gunzip it from the command line, I got the following error:

        gzip: myfile.json.gz: unexpected end of file
def try6_unexpected_end_of_file():

    fgz = cStringIO.StringIO()
    gzip_obj = gzip.GzipFile(
        filename=FILENAME, mode='wb', fileobj=fgz)
    gzip_obj.write(STUFF_TO_GZIP)
    gzip_obj.flush()
    filesize = pycurl_simulator(fgz)
    print filesize

Try 7: got it working

I knew that GzipFile worked properly when writing files directly as opposed to reading from the StringIO object. It turns out the difference was that there was code in the close() method of GzipFile which wrote some extra required data. Now stuff was working.

def try7_got_it_working():

    fgz = cStringIO.StringIO()
    gzip_obj = gzip.GzipFile(
        filename=FILENAME, mode='wb', fileobj=fgz)
    gzip_obj.write(STUFF_TO_GZIP)
    gzip_obj.flush()

    # Do stuff that GzipFile.close() does
    gzip_obj.fileobj.write(gzip_obj.compress.flush())
    gzip.write32u(gzip_obj.fileobj, gzip_obj.crc)
    gzip.write32u(gzip_obj.fileobj, gzip_obj.size & 0xffffffffL)

    filesize = pycurl_simulator(fgz)
    print filesize

Try 8: (not really) final version

Here's the (not really) final version using a subclass of GzipFile that adds a method to write the extra data at the end. If also overrides close() so that stuff isn't written twice in case you need to use close(). Also, the separate flush() call is not needed.

def try8_not_really_final_version():

    class MemoryGzipFile(gzip.GzipFile):
        """
        A GzipFile subclass designed to be used with in memory file like
        objects, i.e. StringIO objects.
        """

        def write_crc_and_filesize(self):
            """
            Flush and write the CRC and filesize. Normally this is done
            in the close() method. However, for in memory file objects,
            doing this in close() is too late.
            """
            self.fileobj.write(self.compress.flush())
            gzip.write32u(self.fileobj, self.crc)
            # self.size may exceed 2GB, or even 4GB
            gzip.write32u(self.fileobj, self.size & 0xffffffffL)

        def close(self):
            if self.fileobj is None:
                return
            self.fileobj = None
            if self.myfileobj:
                self.myfileobj.close()
                self.myfileobj = None

    fgz = cStringIO.StringIO()
    gzip_obj = MemoryGzipFile(
        filename=FILENAME, mode='wb', fileobj=fgz)
    gzip_obj.write(STUFF_TO_GZIP)
    gzip_obj.write_crc_and_filesize()

    filesize = pycurl_simulator(fgz)
    print filesize

Try 9: didn't need to do that (final version)

It turns out I can close the GzipFile object and the StringIO object remains available. So that MemoryGzipFile class above is completely unnecessary. I am dumb. Here is the final iteration:

def try9_didnt_need_to_do_that():

    fgz = cStringIO.StringIO()
    gzip_obj = gzip.GzipFile(
        filename=FILENAME, mode='wb', fileobj=fgz)
    gzip_obj.write(STUFF_TO_GZIP)
    gzip_obj.close()

    filesize = pycurl_simulator(fgz)
    print filesize

References

Here is some googling I did:

How to start a long-running process in screen and detach from it

How to start a long-running process in screen, detach from it, and reattach to it later.

Start a long running process in screen and detach

  • Ssh to the remote host, myremote:
    eliot@mylocal:~$ ssh myremote 
    
  • Start a new screen session
    eliot@myremote:~$ screen 
    
  • Start a long running process, "sleep 3600":
    eliot@myremote:~$ sleep 3600 
    
  • Detach from the screen session:
    eliot@myremote:~$ CTRL-A : detach ENTER 
    
    (Hit [CTRL-A], then type a colon character, then type "detach", then hit [ENTER])
  • Exit your remote SSH session:
    eliot@myremote:~$ exit 
    

Reattach to the existing screen session

  • Ssh to the remote host again:
    eliot@mylocal:~$ ssh myremote 
    
  • List your active screen sessions:
    eliot@myremote:~$ screen -ls 
    There is a screen on:
    	11518.pts-1.myremote	(Detached)
    1 Socket in /var/run/screen/S-eliot.
    
  • Reattach to your screen session:
    eliot@myremote:~$ screen -RD 
    
    Note: you don't actually have to use the -RD option. You could use -rD or -r. But I just use -RD all the time. If there is more than one screen session active you will have to say: screen -RD 11518.pts-1.myremote or whichever screen session you want to attach to.
  • It will show you the "sleep 3600" command running. To exit, CTRL-C the sleep process, type "exit" to exit the screen session, and "exit" again to exit the SSH session.

See also

How to use pip with crate.io

Here's how to use pip with crate.io (in case pypi.python.org goes down):
$ pip install --index-url=https://simple.crate.io yolk 
Or with logging to see what's happening:
$ pip install --log=my-pip-debug.log --index-url=https://simple.crate.io yolk 

See also

How to run a Django local development server on a remote machine and access it in your browser on your local machine using SSH port forwarding

Here is how to run a Django local development server on a remote machine and access it in your browser on your local machine using SSH port forwarding. (This is useful if there is a firewall blocking access to the port of your Django local dev server (port 8000).

  1. On the local host, SSH to the remote host:
    $ ssh -v -L 9000:localhost:8000 [email protected] 
    
  2. On the remote host, run the Django dev server:
    [email protected]:/path/to/my/django/project$ python manage.py runserver 0.0.0.0:8000 
    
  3. On the local host, go to http://localhost:9000 in the browser

Note: The local port and the remote port can be the same (i.e. you can use 8000 instead of 9000). I just made them different to show which port is which.

Using LocalForward in your ~/.ssh/config

You can also achieve the same results by using the LocalForward in your ~/.ssh/config file:

Host myremote
  User eliot
  HostName my.remotehost.com
  LocalForward 9000 localhost:8000

Reference

http://magazine.redhat.com/2007/11/06/ssh-port-forwarding/

Testing HTTPS w/ Flask's development server using stunnel on Ubuntu

Our website is served over HTTPS. To more easily test certain issues (e.g. mixed mode content warnings, or Mapquest SSL tile servers), I wanted to access my Flask local development server over HTTPS. These two articles describe how to do this using stunnel: Testing HTTPS with Django's Development Server, Django Development Server with HTTPS. Using stunnel, you can hit pages on your Django/Flask local dev server over HTTPS instead of HTTP. Here is how I installed it on Ubuntu Precise 12.04:

  • Install SSL development files
    $ sudo apt-get install libssl-dev 
    
  • Go to https://www.stunnel.org/downloads.html and download stunnel-4.54.tar.gz
  • Unpack, compile, install.
    $ tar xvf stunnel-4.54.tar.gz 
    $ cd stunnel-4.54 
    $ ./configure --prefix=/home/saltycrane/lib/stunnel-4.54 
    $ make 
    $ make install 
    NOTE: the make install step asked me a number of questions and created a certificate file at /home/saltycrane/lib/stunnel-4.54/etc/stunnel/stunnel.pem. Accept all the defaults for the certificate information (accurate certificate information isn't needed for this application).
  • Create a stunnel configuration file, /home/saltycrane/lib/stunnel-4.54/etc/stunnel/dev_https:
    pid =
    cert = /home/saltycrane/lib/stunnel-4.54/etc/stunnel/stunnel.pem
    debug = 7
    foreground = yes
    
    [https]
    accept = 7000
    connect = 5000
  • Start stunnel:
    $ /home/saltycrane/lib/stunnel-4.54/bin/stunnel /home/saltycrane/lib/stunnel-4.54/etc/stunnel/dev_https
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: Clients allowed=500
    2012.10.17 17:40:52 LOG5[12468:140357811214080]: stunnel 4.54 on x86_64-unknown-linux-gnu platform
    2012.10.17 17:40:52 LOG5[12468:140357811214080]: Compiled/running with OpenSSL 1.0.1 14 Mar 2012
    2012.10.17 17:40:52 LOG5[12468:140357811214080]: Threading:PTHREAD SSL:+ENGINE+OCSP Auth:none Sockets:POLL+IPv6
    2012.10.17 17:40:52 LOG5[12468:140357811214080]: Reading configuration from file /home/saltycrane/lib/stunnel-4.54/etc/stunnel/dev_https
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: Compression not enabled
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: Snagged 64 random bytes from /home/saltycrane/.rnd
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: Wrote 1024 new random bytes to /home/saltycrane/.rnd
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: PRNG seeded successfully
    2012.10.17 17:40:52 LOG6[12468:140357811214080]: Initializing service [https]
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: Certificate: /home/saltycrane/lib/stunnel-4.54/etc/stunnel/stunnel.pem
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: Certificate loaded
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: Key file: /home/saltycrane/lib/stunnel-4.54/etc/stunnel/stunnel.pem
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: Private key loaded
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: Using DH parameters from /home/saltycrane/lib/stunnel-4.54/etc/stunnel/stunnel.pem
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: DH initialized with 1024-bit key
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: ECDH initialized with curve prime256v1
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: SSL options set: 0x00000004
    2012.10.17 17:40:52 LOG5[12468:140357811214080]: Configuration successful
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: Service [https] (FD=7) bound to 0.0.0.0:7000
    2012.10.17 17:40:52 LOG7[12468:140357811214080]: No pid file being created
    
  • Start the python dev server:
    $ HTTPS=1 python bin/runserver.py 0.0.0.0 5000 
  • Go to https://localhost:7000 in your browser

See also

Python data object motivated by a desire for a mutable namedtuple with default values

UPDATE 2016-08-12: Read Glyph's post and use the attrs library instead.

Reasons to use this instead of a namedtuple:

  • I want to change fields at a later time (mutability)
  • I want to specify a subset of the fields at instantiation and have the rest be set to a default value

Reasons to use this instead of a dict:

  • I want to explicitly name the fields in the object
  • I want to disallow setting fields that are not explicitly named*
  • I want to specify a subset of the fields at instantiation and have the rest be set to a default value
  • I want to use attribute style access (dot notation to access fields)

Reasons to use this instead of a regular Python class:

  • I don't want to duplicate field names in the __init__() method signature and when setting instance attributes of the same name.
  • I want to disallow setting fields that are not explicitly named*
  • I want to be able to easily convert the object to a dict or a tuple
  • I want to save memory

*Note: This Stack Overflow answer warns against using __slots__ for my goal of disallowing setting fields that are not explicitly named. It says metaclasses or decorators should be abused by us control freaks and static typing weenies instead. To comply with that advice, if you don't care about saving memory, __slots__ could be replaced with a non-special attribute, such as _fields. If that is done, attribute creation would no longer be limited.

See also:

class DataObject(object):
    """
    An object to hold data. Motivated by a desire for a mutable namedtuple with
    default values. To use, subclass, and define __slots__.

    The default default value is None. To set a default value other than None,
    set the `default_value` class variable.

    Example:
        class Jello(DataObject):
            default_value = 'no data'
            __slots__ = (
                'request_date',
                'source_id',
                'year',
                'group_id',
                'color',
                # ...
            )
    """
    __slots__ = ()
    default_value = None

    def __init__(self, *args, **kwargs):
        # Set default values
        for att in self.__slots__:
            setattr(self, att, self.default_value)

        # Set attributes passed in as arguments
        for k, v in zip(self.__slots__, args):
            setattr(self, k, v)
        for k, v in kwargs.items():
            setattr(self, k, v)

    def asdict(self):
        return dict(
            (att, getattr(self, att)) for att in self.__slots__)

    def astuple(self):
        return tuple(getattr(self, att) for att in self.__slots__)

    def __repr__(self):
        return '{}({})'.format(
            self.__class__.__name__,
            ', '.join('{}={}'.format(
                    att, repr(getattr(self, att))) for att in self.__slots__))

Tests:

import unittest

class DataObjectTestCase(unittest.TestCase):
    def test_instantiation_using_args(self):
        class MyData(DataObject):
            __slots__ = ('att1', 'att2')

        md = MyData('my attr 1', 'my attr 2')
        self.assertEqual(md.att1, 'my attr 1')
        self.assertEqual(md.att2, 'my attr 2')

    def test_instantiation_using_kwargs(self):
        class MyData(DataObject):
            __slots__ = ('att1', 'att2')

        md = MyData(att1='my attr 1', att2='my attr 2')
        self.assertEqual(md.att1, 'my attr 1')
        self.assertEqual(md.att2, 'my attr 2')

    def test_default_default_value(self):
        class MyData(DataObject):
            __slots__ = ('att1', 'att2')

        md = MyData(att1='my attr 1')
        self.assertEqual(md.att1, 'my attr 1')
        self.assertEqual(md.att2, None)

    def test_custom_default_value(self):
        class MyData(DataObject):
            default_value = 'custom default value'
            __slots__ = ('att1', 'att2')

        md = MyData(att1='my attr 1')
        self.assertEqual(md.att1, 'my attr 1')
        self.assertEqual(md.att2, 'custom default value')

    def test_set_value_after_instantiation(self):
        class MyData(DataObject):
            __slots__ = ('att1', 'att2')

        md = MyData(att1='my attr 1')
        self.assertEqual(md.att1, 'my attr 1')
        self.assertEqual(md.att2, None)
        md.att1 = 5
        md.att2 = 9
        self.assertEqual(md.att1, 5)
        self.assertEqual(md.att2, 9)

    def test_attribute_not_defined_in__slots__(self):
        class MyData(DataObject):
            __slots__ = ('att1', 'att2')

        with self.assertRaises(AttributeError):
            MyData(att3='my attr 3')

        with self.assertRaises(AttributeError):
            md = MyData()
            md.att3 = 45

    def test_asdict(self):
        class MyData(DataObject):
            __slots__ = ('att1', 'att2')

        md = MyData(att1='my attr 1', att2='my attr 2')
        self.assertEqual(
            md.asdict(), {'att1': 'my attr 1', 'att2': 'my attr 2'})

    def test_tuple(self):
        class MyData(DataObject):
            __slots__ = ('att1', 'att2')

        md = MyData(att1='my attr 1', att2='my attr 2')
        self.assertEqual(md.astuple(), ('my attr 1', 'my attr 2'))

    def test___repr__(self):
        class MyData(DataObject):
            __slots__ = ('att1', 'att2')

        md = MyData(att1='my attr 1', att2='my attr 2')
        self.assertEqual(repr(md), "MyData(att1='my attr 1', att2='my attr 2')")

Note: previously, I included the following method in the class. However, this is not necessary. If __slots__ is defined in DataObject and the subclass, any attribute not in __slots__ will automatically raise an AttributeError.

#     def __setattr__(self, name, value):
#         if name not in self.__slots__:
#             raise AttributeError("%s is not a valid attribute in %s" % (
#                     name, self.__class__.__name__))
#         super(DataObject, self).__setattr__(name, value)

Setting up a Linux DVR w/ MythTV, Ubuntu 12.04, and a Hauppauge WinTV-HVR 1250 TV tuner card

Setting up MythTV involves a little pain, but once it's set up, it's pretty great. And you don't have to spend lots of money on a DVR from the cable company. With my modest hardware specs, playback is smooth and clear, however Picture in Picture is too jittery to be useful. Here's what I did to get my MythTV DVR running on my Ubuntu machine.

Parameters

Install the Hauppauge WinTV-HVR 1250 TV tuner card

Put the card in the computer. Connect the TV antenna to the card.

Check the TV tuner card is recognized

Ubuntu 12.04 includes drivers for the Hauppauge 1250 TV tuner card, so I did not need to install any drivers.

Update 2016-10-16:On Ubuntu 16.04, look in /var/log/syslog instead of /var/log/dmesg.

$ cat /var/log/dmesg
[   15.211985] cx23885 driver version 0.0.3 loaded
[   15.214279] cx23885 0000:03:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[   15.214492] CORE cx23885[0]: subsystem: 0070:2259, board: Hauppauge WinTV-HVR1255 [card=20,autodetected]
[   15.214600] IR NEC protocol handler initialized
[   15.230936] IR RC5(x) protocol handler initialized
[   15.235576] MCE: In-kernel MCE decoding enabled.
[   15.237132] IR RC6 protocol handler initialized
[   15.237703] EDAC MC: Ver: 2.1.0
[   15.238256] AMD64 EDAC driver v3.4.0
[   15.242493] IR JVC protocol handler initialized
[   15.246743] IR Sony protocol handler initialized
[   15.250908] IR MCE Keyboard/mouse protocol handler initialized
[   15.256862] lirc_dev: IR Remote Control driver registered, major 250 
[   15.257125] IR LIRC bridge handler initialized
[   15.284735] lp0: using parport0 (interrupt-driven).
[   15.361892] tveeprom 0-0050: Hauppauge model 22111, rev E2F5, serial# 8323201
[   15.361895] tveeprom 0-0050: MAC address is 00:0d:fe:7f:00:81
[   15.361897] tveeprom 0-0050: tuner model is NXP 18271C2 (idx 155, type 54)
[   15.361899] tveeprom 0-0050: TV standards NTSC(M) ATSC/DVB Digital (eeprom 0x88)
[   15.361901] tveeprom 0-0050: audio processor is CX23888 (idx 40)
[   15.361903] tveeprom 0-0050: decoder processor is CX23888 (idx 34)
[   15.361904] tveeprom 0-0050: has no radio, has IR receiver, has no IR transmitter
[   15.361906] cx23885[0]: hauppauge eeprom: model=22111
[   15.361909] cx23885_dvb_register() allocating 1 frontend(s)

Install MythTV

$ sudo apt-get install mythtv

Set up the MythTV backend

Run mythtv-setup to select your TV tuner card and scan for channels.

$ mythtv-setup

Click "Yes" to add your user to the "mythtv" group.

Click "Yes" to restart your login session.

Change the following options:

  • 2. Capture cards -> (New capture card) -> Card type: DVB DTV capture card (v3.x) -> Finish
  • 4. Video sources -> (New video source) -> Video source name: FOOBAR, Listings grabber: North America (SchedulesDirect.org) (Internal), User ID: blank, Pass: blank
  • 5. Input connections -> [DVB: /dev/dvb/adapter0/frontend0] -> Video source: FOOBAR -> Scan for channels
    Note: if you see the error "Failed to open the card", make sure the mythv-backend is actually stopped. The GUI said it would stop the backend but it was still running.

After running mythtv-setup, it will ask you if you want to start the backend. Select yes to start the backend. It will also ask you if you want to run mythfilldatabase. Select yes to run mythfilldatabase. This may take a while.

Ensure mythv backend is running

After running mythtv-setup, the mythtv backend should start running.

To check that the backend is running, run:

$ ps -ef | grep myth

If the mythtv backend is not running, start it using the following command:

$ sudo service mythtv-backend start

Troubleshooting mythbackend

If mythbackend doesn't stay running, there may be some configuration that is broken. Check /var/log/syslog. If that does not have enough information, run the backend with the --verbose option:

$ mythbackend --verbose

Run the MythTV frontend

$ mythfrontend

Some keyboard shortcuts

  • P - pause/play
  • SPACE - set/clear bookmark
  • LEFT/RIGHT ARROW - skip back/forward
  • M - menu
  • D - delete

Other stuff

  • You may want to change the theme. I chose the TintedGlass 2.43 theme.
  • To get schedule information, I ended up signing up for a membership at www.schedulesdirect.org. It is $25/year (or ~$2/month). It seems to be the recommended way to get schedule information.

How run mythfrontend on another Ubuntu laptop connected to your LAN (Added 2013-06-07)


Since MythTV has a flexible client/server architecture, you can run the MythTV backend server on one machine and access it from multiple other machines running a Mythtv frontend. These steps assume the remote frontend is running on a laptop with Ubuntu 12.04 and it is connected to your local network (LAN) (not through the internet (though that is possible.).)

UPDATE: Playing 1080p HD content over my $30 Belkin G wireless router (rated at 54 Mbps) had occasional stalls in the playback. Repositioning my router helped, but after a couple days, I decided to order a Netgear N600 Wireless-N Dual Band Router. Hopefully this will solve my problem.

On the Mythtv backend server configured above:

  • Determine the IP address of the Mythtv backend server by running ifconfig
    $ ifconfig 
    For me, it is 192.168.2.2. This will be used in the steps below.
  • Follow the instructions here: http://www.mythtv.org/wiki/Mythfrontend
    • Edit /etc/mysql/my.cnf so the the bind-address line is commented out (on Ubuntu 16.04 edit /etc/mysql/mysql.conf.d/mysqld.cnf):
      #bind-address 127.0.0.1
    • Allow remote users access to the database. Note: replace "mypassword" with the value found in ~/.mythtv/mysql.txt.
      $ mysql -u root
      mysql> grant all on mythconverg.* to 'mythtv'@'%' identified by 'mypassword';
      mysql> flush privileges;
      mysql> exit
    • Restart mysql server:
      $ sudo service mysql restart 
  • Ensure mythbackend is not using 127.0.0.1.
    • Run mythtv-setup:
      $ mythtv-setup 
    • Change the IP address from 127.0.0.1 to 192.168.2.2 (or IP address you determined from above.)

On the laptop:

  • $ sudo apt-get install mythtv-frontend 
  • click "yes" to be added to the mythtv group
  • click "yes" to restart your session
  • click "OK" to the msg about logging out of your session
  • logout and login again
  • Run mythfrontend
    $ mythfrontend 
    
  • For the hostname: enter the IP address of the Mythtv server. For me it is 192.168.2.2.
  • Enter the Mysql password. This can be found in ~/.mythtv/mysql.txt (or ~/.mythtv/config.xml or /etc/mythtv/config.xml on newer versions) on the Mythtv server machine. Or you can check the settings of the mythfrontend running on the server machine.

How to watch your recorded videos on your Android phone over the internet


  • This method uses the MythTV Services API
  • PC: Set up a SSH server on your MythTV backend server
  • PC: Get the external IP address of your MythTV backend server
    $ curl http://ifconfig.me
    111.222.333.444 
  • Android: Install Connectbot on your Android phone and enable port forwarding of 6544. For more info see: http://parker1.co.uk/mythtv_ssh.php
    • Android: Using Connectbot, connect to your MythTV server using the IP address from above (111.222.333.444)
    • Android: Menu -> Port Forwards -> Menu -> Add port forward:
      • Nickname: mythtv
      • Type: local
      • Source port: 6544
      • Destination: localhost:6544
    • Android: Disconnect and reconnect
  • Android: Install and set up MythTV Android Frontend
    • Android: Touch the settings icon -> Away Profiles
      • Name: Away
      • MythTV Master Backend Address: http://localhost:6544/
      Save
    • Android: Away -> Recordings -> Select a show to watch -> watch it

See also:

How to control your DVR from your Android phone

  • Configure your MythTv Frontend on your PC:
    • Setup -> General -> Hit "Next" 6 times -> Check "Enable Network Remote Control"
    • Setup -> Appearance -> Hit "Next" 3 times and
      • Check "Enable LCD device"
      • Check "Display time"
      • Check "Display menus"
      • Check "Display music arstist and title"
      • Check "Display channel information"
  • Install MythDroid on your Android phone
  • Install MDD on your PC
    • Install libimlib2
      $ sudo apt-get install libimlib2-dev 
      
    • Download MDD
      $ wget http://mythdroid.googlecode.com/files/mdd-0.6.2.tgz 
      
    • Install MDD
      $ tar xvf mdd-0.6.2.tgz 
      $ cd mdd 
      $ perl Build.PL 
      
      Type "y" because you are running this on the PC that runs your MythTv frontend
      $ ./Build installdeps 
      
      Hit ENTER to accept all the defaults
      $ ./Build test 
      $ sudo ./Build install 
      
      Type "y" to stop mythfrontend. Then start it again
      $ mythfrontend 
      

How to restore an old database on a fresh install Added 2016-10-16

See https://www.mythtv.org/wiki/Database_Backup_and_Restore

$ wget https://raw.githubusercontent.com/MythTV/mythtv/master/mythtv/programs/scripts/database/mythconverg_restore.pl 
$ chmod a+x mythconverg_restore.pl 
$  ./mythconverg_restore.pl --directory /mnt/hdd2/mythtv/db_backups --filename mythconverg-1317-20161008080149.sql.gz --drop_database --create_database 
$ # if there has been a database schema change, you will need to run mythtv-setup

Help / References