Python snippets

Some snippets of Python code that I use too infrequently to remember, but frequently enough to be annoyed every time I have to track down an example again.

iPython3 execfile helper

Python3 removes execfile, so its nice to have a helper.

Simple:

coder = lambda x : compile(open(x).read(), x, 'exec')

Use it like so:

exec(coder(filename))

More complex including setting the global __file__ variable to the executed filename (which can sometimes be useful):

coder = lambda x : [compile(open(x).read(), x, 'exec'), globals().update({'__file__' : x})]

Use:

exec(coder(filename))

Put these in ~/.ipython/profile_default/startup/startup.py to make them always available in iPython.

Generate sortable timestamp

Do the necessary import

import datetime

Timestamp with only the date:

def timestamp():
    return datetime.datetime.now().strftime('%Y%m%d')

Timestamp with date and time:

def timestamp():
    return datetime.datetime.now().strftime('%Y%m%d%H%M%S')

Timestamp to datetime and reverse

Do the import

from datetime import datetime

Timestamp to datetime

datetime.fromtimestamp(1545730073)

Datetime to timestamp

datetime.timestamp(datetime.now())

An example HTML parser

An example of a HTMLPaser, you will need to edit this to make it useful, this is just a template thats a bit better than the obvious one in the Python docs.

from html.parser import HTMLParser
class HParser(HTMLParser):
    '''Example HTML parser'''

    def __init__(self):
        HTMLParser.__init__(self)
        self.outData = ''
        self.divData = ''
        self.inHtag = False
        self.inDtag = False


    def handle_starttag(self, tag, attrs):
        if tag == 'h1':
            self.inHtag = True
        elif tag == 'div':
            if (self.get_starttag_text().find('content') > -1):
                self.inDtag = True
    
    def handle_endtag(self, tag):
        if tag == "h1":
            self.inHtag = False
        elif tag == "div":
            self.inDtag = False


    def handle_data(self, data):
        if self.inHtag:
            self.outData = self.outData + data
        elif self.inDtag:
            self.divData = self.divData + data
            

    def close(self):
        return [ self.outData, self.divData ]

HTML parsing using BeautifulSoup

This simple example shows how to get a list of all “td” elements in HTML in the variable html_text.

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_text, 'html.parser')
soup.find_all('td')

wsgi example to load a Flask app

Most of the times my Python helper servers are single use code and run temporarily listening on their own socket, but sometimes you want to offer multiple functions and take advantage of existing infrastructure, such as an existing https server with legitimate certificate. In this case, the following with the wsgi module will allow a web server like Apache to serve the Python code.

# if the app isn't already installed as a module or in the Python path, import sys and add the folder it sits in to Python path
import sys
sys.path.insert(0, '/usr/local/www/wsgi-scripts/')

# where flask_app is the name of the script file/module
from flask_app import app as application

A good reference for the Apache configuration is here, which can be followed once the wsgi module is installed and enabled using a2enmod. The WSGIScriptAlias and Allow/Require instructions (depending on version) are minimums, for SSL you may also want to add SSLOptions +StdEnvVars.

An example Flask app that could be loaded is here.

A simple Python3 https server

From Github gists here.

Python http command

python -m http.server 8000

Python smtp stdout email receiver command

sudo python -m smtpd -n -c DebuggingServer 0.0.0.0:25

Python __import__

An example of calling the path.isdir method from the os module without an explicit import. Full docco here.

__import__('os').path.isdir('/Users')

Xor

Xor with a single character:

def xorSingle(input: bytes, xorVal: bytes) -> bytes:
    return bytes([a^xorVal[0] for a in input])

Xor two values, truncate to the length of the shortest:

def xorValues(input1: bytes, input2: bytes) -> bytes:
    return bytes([input1[a] ^ input2[a] for a in range(0,min(len(input1),len(input2)))])

Xor one longer value with a repeating shorter second value:

def xorRepeat(input1: bytes, xorVal: bytes) -> bytes:
    return bytes([input1[a] ^ xorVal[a%len(xorVal)] for a in range(0,len(input1))])

Crypto

PCKS7 padding

def pad(value: bytes, bs: int=16) -> bytes:
    pv = bs - (len(value) % bs)
    return value + (chr(pv) * pv).encode()

PKCS7 unpadding

def unpad(value: bytes, bs: int=16) -> bytes:
    pv = value[-1]
    if pv > bs:
        raise Exception('Bad padding')
    padding = value[-pv:]
    if len(padding) != pv or len(set([a for a in padding])) != 1:
        raise Exception('Bad padding')
    return value[:-pv]

AES CBC encrypt and decrypt with leading IV and PKCS7 padding

from Crypto.Cipher import AES
from Crypto import Random
BLOCKSIZE=16

def encrypt(value: bytes, key:bytes) -> bytes:
    iv = Random.new().read(BLOCKSIZE)
    cipher = AES.new(key, AES.MODE_CBC, iv) 
    return iv + cipher.encrypt(pad(value))

def decrypt(value: bytes, key: bytes) -> bytes:
    cipher = AES.new(key, AES.MODE_CBC, value[:BLOCKSIZE]) # iv is value[:BLOCKSIZE]
    decrypted = cipher.decrypt(value[BLOCKSIZE:]) # encrypted data is value[BLOCKSIZE:]
    return unpad(decrypted)

Run executable and get output

A very simple approach

import subprocess
subprocess.check_output(['ls', '-lart']).decode()

Or a slightly more involved example:

def run_process(command: list) -> str:
    proc = subprocess.Popen(command, stdout=subprocess.PIPE)
    out =  proc.stdout.read().decode()
    proc.terminate()
    return out

Resolve a name using DNS

import dns.resolver

def resolve(name, rdtype='A'):
    return [a.to_text() for a in dns.resolver.resolve(name, rdtype=rdtype)]

Adding error handling to prevent exceptions in case of lookup failures:

def try_resolve(name, rdtype='A'):
    try:
        return resolve(name, rdtype)
    except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer, dns.resolver.NoNameservers):
        return []

Test if you can connect to a host on a given TCP port

import socket


def test_connect(host: str, port: int) -> bool:
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.settimeout(1)
    result = sock.connect_ex((host, int(port)))
    return True if result == 0 else False

Get the certificate from a remote SSL service

import socket
from OpenSSL import SSL

def get_cert(hostname: str, port: int):
    sock = socket.socket()
    sock.connect((hostname, int(port)))
    ctx = SSL.Context(SSL.SSLv23_METHOD) # most compatible
    ctx.check_hostname = False
    ctx.verify_mode = SSL.VERIFY_NONE

    sock_ssl = SSL.Connection(ctx, sock)
    sock_ssl.set_connect_state()
    sock_ssl.set_tlsext_host_name(hostname.encode())
    sock_ssl.do_handshake()
    cert = sock_ssl.get_peer_certificate()
    sock_ssl.close()
    sock.close()

    return cert

Create an NTLM hash

import hashlib
ntlm = lambda pwd : hashlib.new('md4', pwd.encode('utf-16le')).hexdigest()
ntlm('password_to_hash')

On newer Pythons that dont include md4 in hashlib, install pycryptodome and try this

from Crypto.Hash import MD4
ntlm = lambda pwd : MD4.new(pwd.encode('utf-16le')).hexdigest()

Get all permutations of a given input list

The following will get all combinations of the given list, including combinations of items of list length n-1 to 2.

from itertools import permutations
getListPerms = lambda x: [ a for b in range(1, len(x)) for a in permutations(x, b+1)]
getListPerms(['one', 'two', 'three', 'four'])

Make the requests module shut up about insecure requests

If you choose to make unverified https requests and the certificate of the site is not considered valid, the requests module makes sure to tell you about it with a InsecureRequestWarning error, saying Unverified HTTPS request is being made to host. This seems to work to make those errors go away:

requests.packages.urllib3.disable_warnings()

Custom argument parser

The custom argument parser I use for Python code that I intend for other people to be able to use to provide a more friendly interface.

Setup:

import sys
import argparse

class MyParser(argparse.ArgumentParser):
    def error(self, message):
        sys.stderr.write('error: %s\n' % message)
        self.print_help()
        sys.exit(2)

Example use:

if __name__ == "__main__":
    parser = MyParser()
    parser.epilog = 'Text to include after help output'
    parser.description = 'Text to include after usage and before options help'
    parser.add_argument('-c', '--cookies', type=str, required=True, help='Cookie value. REQUIRED.')
    parser.add_argument('-r', '--role', type=str,  default='role1', help='Help string. Default: role1')
    parser.add_argument('-o', '--other', action='store_true', help='Not implemented')
    args = parser.parse_args()
    cookie = args.cookies

Custom logger

The custom logger I use for more significant Python programs.

Setup:

import logging
from logging import Logger 

def create_logger(loglevel: str, name: str) -> Logger:
    logger = logging.getLogger(name)
    logger.setLevel(loglevel)
    handler = logging.StreamHandler(sys.stderr)
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    handler.setFormatter(formatter)
    logger.addHandler(handler)

    return logger


logger = create_logger(args.loglevel, 'Log Name')

Usage:

logger.info('Info message')

Check if code is running in iPython

Useful to add to scripts to enable you to determine whether it is running in iPython or not to enable different code flows

def check_ipython():
    """Returns True if script is running in interactive iPython shell"""
    try:
        get_ipython()
        return True
    except NameError:
        return False