Two big features I’m interested in and I’ll cover today:
Preface:
Assuming you already have working k8s cluster and context, you need to install the TS Operator. Installing via Helm is also an option, but I chose static files.
Get the latest development version:
wget https://raw.githubusercontent.com/tailscale/tailscale/main/cmd/k8s-operator/deploy/manifests/operator.yaml
Create a new TS Oauth app, for the Operator: https://login.tailscale.com/admin/settings/oauth
Then configure the OAuth secret in the file:
client_id: XXXXXXXXXX
client_secret: tskey-XXXXX
I’m also also choosing to use the TS proxy in auth mode, which “impersonates requests from tailnet to the Kubernetes API server”. This requires some additional configuration, but seems a bit more flexible versus the “noauth” mode, which only proxies requests and requires a bit more lower-level work to have your TLS certs configured correctly.
Configure the Operator section in the file:
name: APISERVER_PROXY
value: "true"
Apply the file to your cluster
$ k apply -f operator.yaml
namespace/tailscale created
secret/operator-oauth created
serviceaccount/operator created
serviceaccount/proxies created
customresourcedefinition.apiextensions.k8s.io/connectors.tailscale.com created
customresourcedefinition.apiextensions.k8s.io/proxyclasses.tailscale.com created
clusterrole.rbac.authorization.k8s.io/tailscale-operator created
clusterrolebinding.rbac.authorization.k8s.io/tailscale-operator created
role.rbac.authorization.k8s.io/operator created
role.rbac.authorization.k8s.io/proxies created
rolebinding.rbac.authorization.k8s.io/operator created
rolebinding.rbac.authorization.k8s.io/proxies created
deployment.apps/operator created
ingressclass.networking.k8s.io/tailscale created
Then for auth proxy, apply the RBAC roles in the most insecure fashion:
curl -s https://raw.githubusercontent.com/tailscale/tailscale/main/cmd/k8s-operator/deploy/manifests/authproxy-rbac.yaml | k apply -f -
clusterrole.rbac.authorization.k8s.io/tailscale-auth-proxy unchanged
clusterrolebinding.rbac.authorization.k8s.io/tailscale-auth-proxy unchanged
I’m using my own private tailscale network, so auth-ing myself with all privileges is easiest:
kubectl create clusterrolebinding askedrelic --user="askedrelic@github" --clusterrole=cluster-admin
Lastly, you can use Tailscale to create your K8S context and use it!
tailscale configure kubeconfig tailscale-operator
You should have K8S access using the TS context now!
There are other options for getting secure access to your apiserver via ssh forwarding or tailscale subnets on the server itself, but consolidating more configuration like this into K8S makes sense for me.
Lets run an app (only internally available to K8S) and connect to it over Tailscale. I’ll use the default nginx image:
k create deployment nginx --image=nginx
There are several options TS exposes. The first is creating via a Service:
kind: Service
apiVersion: v1
metadata:
name: nginx-ts
spec:
type: LoadBalancer
loadBalancerClass: tailscale
selector:
app: nginx
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
Assuming that creates successfully, you can get the Service details:
$ k get service nginx-ts
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ts LoadBalancer 10.96.207.167 100.121.9.15,default-nginx-ts-2.bee-hake.ts.net 80:30697/TCP 9s
And it should be available via the TS IP:
$ curl 100.121.9.15
...
<h1>Welcome to nginx!</h1>
...
or generated TS device name, assuming you have TS DNS enabled locally:
$ curl default-nginx-ts-2.bee-hake.ts.net
The other option is via Ingress, which can make the app available via HTTPS with a Lets Encrypt generated cert, assuming you have HTTPS enabled in your TS settings.
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ts
spec:
defaultBackend:
service:
name: nginx
port:
number: 80
ingressClassName: tailscale
---
apiVersion: v1
kind: Service
metadata:
name: nginx-ts
spec:
ports:
- name: http
port: 80
targetPort: 80
selector:
app: nginx
type: ClusterIP
While testing this, Service/Ingress changes didn’t seem to be reconciled by the Operator, so delete/re-creating seems required for now.
And now it should be available via HTTPS:
$ curl https://default-nginx-ingress.bee-hake.ts.net
...
<h1>Welcome to nginx!</h1>
...
Overall, the Operator has worked smoothly for me in testing and it has made things more convenient and secure for my network.
In Part 2, I’ll explore connecting apps together over the TS network.
]]>Several other printable calendars I’ve found recently, that were inspirational:
But I couldn’t find a “quarter” or 12 week view to cover my Recurse timeframe!
My first thought was to stick with a static HTML+CSS page, but I decided to have the dates be dynamic, which requires Javascript. And once you start doing any kind of complex Javascript, it’s convenient to use React, inline CSS styling with your React components, and then it turned into a fully compiled project. Therefore, I chose,
Hugo was easy to learn and get started with. It has hot-reload support, which is important for HTML design work. I initially tried to use static <script>
tags for importing React and Tailwind, but moved to a fully imported and compiled pipeline all inside Hugo, to get importing and tree shaking working. Hugo uses esbuild, which seems like a really nice optimal asset compiler.
Two links that helped me understand the Hugo asset pipeline
At this point, I had compiled HTML and Javascript, but CSS was still being included via a static script tag in the HTML header:
<script src="https://cdn.tailwindcss.com"></script>
I wanted to inline importing Tailwind CSS, so that I could use it inside my React code. Tailwind recommends their own separate watch and compile step, like so:
npx tailwindcss -i ./src/input.css -o ./src/output.css --watch
But I mostly ignored this. I ran into caching issues in dev and hot-reloading, where 50% of the time a Hugo rebuild would not pickup my Tailwind CSS changes in a JS file. I was mostly done with the project, so I didn’t investigate further, since a full hugo build always worked. I did find several tickets with suggested fixes.
The last visual debugging I did was around how the actual printed product looked! I printed several copies, as I was working, and tweaked some of the font sizing and colors for clarity. It leaves a nice paper trail of prototypes:
Here is the finished product: https://quarterly-cal.asktherelic.com
Overall, bootstrapping and developing the app were pretty straightforward; I’ve done enough HTML+CSS before. Tailwind is a good standard framework for CSS that makes sense for me.
The hardest part was the Javascript datemath for calculating and rendering out the week numbers and spacing. I was trying to not import too many libraries, but not using a JS date library made this more difficult.
Source: https://github.com/askedrelic/quarterly-calendar/
]]>
I’m fortunate to be in a stable financial position, allowing me to step outside my comfort zone and explore something new. While my career has been built on predictability and meeting expectations, I believe there’s value in venturing outside your comfort zone, especially when it aligns with your true self. My hope is to discover new passions and delve into them more deeply than I have lately. This experience brings back memories of my college days when I lived on a “special interest” dormitory floor for computers, called Computer Science House, that had a great community and environment for learning. Recurse Center reminds me of that fondly.
The Recurse Center principles prioritize self-direction and working towards personal goals. So, what are my goals?
Even drafting this blog post contributes to my objectives. Over the last few years, I’ve felt a creative block as my sharing has been confined to internal work discussions. In an introduction meeting, I heard the phrase, “the community is a resource to contribute to and draw from.” This resonates with me, and I’m eager to share my experiences on my blog and contribute to the larger community.
My broad goals include:
Striking the right balance between challenge and focus is a work in progress, but every day is progress. I’m excited about what I plan to accomplishment during my time at the Recurse Center and look forward to the upcoming weeks.
]]>Dokku is a free replacement for Heroku, that you can run on your own server. Without using too many acronyms, it’s an app management tool built around Docker. I started using Heroku back in 2011, when they added support for Python. The simple command line interface for app deployment and free CPU time was a great selling point.
But as costs lowered for servers and with a personal desire to control more of my own tools, for privacy and security reasons, the option to run my own became really interesting.
There have been three major versions of my blog, with different goals at different times (although using three different programming languages wasn’t the goal, just a coincidence):
Middleman is still currently being used to generate this blog as a static website, but now with Dokku, it can be deployed and updated via git push, which is really easy and convenient!
Docker is new hotness that makes deploying code really easy, but it’s still a very manual tool, which is where Dokku comes in. Dokku controls Docker and makes it easy to deploy whole applications; connecting databases, caching servers, or multiple servers together easily. It has a a simple command line interface and support for plugins.
So what this means to me:
So I recommend Dokku if you are interested in running webapps easily and controlling your server. You can see this blog’s code on github, if you want to learn more.
]]>I’ve been writing Python for about 8 years now, mostly on a smaller scale, but the last few years at Yelp have been really interesting to see testing done at a larger scale. Testing has become really important to me, as it helps all the other pieces of your software fit together better.
It was great to be able to share what I’ve learned and brush up on my presentation skills. Unfortunately, I didn’t manage to record a video of the actual presentation, but that was also a lesson learned for when you are presenting.
Testing is a best practice for delivering reliable software, but it can be a hard subject when starting out. What should you test and why? How much testing is enough? So you spent three days and wrote out tests for everything in your module, but was that an effective use of your time?
This talk will give an overview of the different layers that you can write tests for and why you should have them. You start with unit tests, mix in some integration tests, and cover with acceptance tests. Sprinkle with specific testing tools to taste. Tools we’ll discuss include py.test, docker, behave, tox, and coverage. Although the talk focus will be on web apps, the ideas will be relevant to all Python applications.
Writing quality tests is important: flaky tests will cost more time than they save and filler tests that don’t test important areas will weigh you down over time. With stable and effective tests for all layers, you build code you can trust, that you can refactor quickly or change easily without breaking everything. It’s as easy as cake!
]]>For the longest time, my home directory (/home/askedrelic
on most systems) has
been a git repo. This has mostly worked but has several problems:
.gitignore
.ssh
folder could get added with your private keysDespite these complaints, this method has worked out me for several years. It has allowed me to use default git functionality (git-submodules) for my vim plugins and easily keep them up to date. I can git-pull and have the latest configurations by re-opening my shell.
I wanted a way to upgrade my existing repo with minimal changes and keep my git-submodules. Several of the Github recommended dotfile repos were not interesting to me for their forced use of ZSH or complex Ruby/Rake scripts to handle updating. I have a bit of sunk cost with Bash and wanted the option to gradually upgrade to ZSH.
Therefore Zach Holman’s dotfiles looked best to me:
It took a few tries to figure out copying his script/bootstrap
was mainly what
I wanted. Moving my existing git-submodules to a new location was obtusely hard,
until I found a script to handle it:
https://github.com/iam-TJ/git-submodule-move.
The upgrade was basically:
git clone https://github.com/askedrelic/dotfiles/ .dotfiles
cd .dotfiles && script/bootstrap
This new layout allows for much better organization going forward. Check it out here and see your dotfiles could use an upgrade: https://github.com/askedrelic/dotfiles/
]]>I have a couple services that I use for my documents, my writing, my photos, and everything else that is a file. Files are easiest to deal with, as long you have them organized easily. You always want your files in two places (your current computer and online), since you will drop your laptop and your current HDD will fail randomly. You want your files in a simple file format that doesn’t depend on a specific application: usually text, lossless image (PNG), or PDF. PDF might be a terrible format for technical reasons, but it’s a great way to preserve websites or Word documents exactly how they currently are. Every application has “Print as PDF”.
Dropbox is the natural choice for most temporary files. You can get 2GB to 5GB free starting out, it is constantly backed up when you are connected to the internet, and instantly available across all your devices. I keep most temporary files in my Dropbox folder, until I can organize them by date and archive them in a more permanent way.
A more permanent home solution for large file backup is an external HDD or NAS. I finally invested in a consumer NAS last year, the Synology Diskstation DS213 for about $200. Drobo is a another well known consumer NAS, but way too expensive in my opinion, especially considering how cheap hardware has become. The Synology has really impressed me in ease of use, software quality, and overall value. Dual disk RAID support, gigabit ethernet, and “app” support with automatic AWS Glacier backup are some of the more technical features that really make it impressive. Here is another good review, from the Wirecutter.
To balance everything out, you want permanent offsite backups: Backblaze or Crashplan are both good. I’ve used Backblaze for years at work, but am considering Crashplan personally, since Backblaze does not support backing up NAS. These programs are similar to Dropbox, but offer a longterm backup for around $50/year, which is a great price for safety and value of an offsite backup.
Now that you have someplace to backup files, backup every important online service you used last year into a simple file format.
Another project I started last year was backing up my childhood photo albums and VHS tapes! Well, I called my mom and asked her to mail them to me, but it’s a first step. How many boxes of tapes or old photographs do you have at your parent’s house? Convert them to a digital format today, before they fall apart.
I’ve looked at local video conversion places and have been quoted $25/tape for VHS to digital conversions, which seems pricey. I think I will wait to find a VCR and manually convert the videos myself with an Elgato USB recorder, which seems to offer a good value/quality trade off.
That’s my recommendation for the year. Everything that is physical fades eventually and everything that is digital can instantly disappear if you aren’t careful. Understand the digital world and do what is necessary to preserve your important memories and documents!
]]>Testing can be approached from many different ways, with different goals. At the unit test level, you are essentially testing your API:
This what many developers think tests are. I have always felt these were important from an academic standpoint, but definitely never gave them their due respect. Unit tests are a great way to explore your code and reflect whether it still makes sense on a second glance. No matter how trivial or simple the code may be, having the ability to change your mind repeatably, with verified results, is useful.
Moving to a higher level, functional or system tests are good to determine if 3rd-party libraries and your application as a whole is working correctly, and usually where I have spent much of my time from a return on time investment. Either the system works or it doesn’t and this can be traced down quickly. This is something I have begun to focus less on, due to getting better gains from improved unit testing. This is basically the debate over top-down versus bottom-up design and I think testing at both ends of the spectrum is important.
This point has been one of the bigger breakthroughs I’ve made: tests communicate the spec. When I’m building a feature, I usually iteratively write code until I say it works. I’m continually running the code and I am doing the evaluating of the output to determine that the computer is generating correct output. But moving that evaluation step into into code and automating is a big, very useful, step. This is essentially BDD; letting business people write a code spec and having to match that spec.
Once a feature has been communicated into a code spec, changing that feature later becomes a migration; not having to start from scratch to ensure all my assumptions still work with subtle changes.
As my team has gotten larger, being able to say my code does something and then have a test to prove it helps with async communication and increases the speed of integration. Having a second source of code truth keeps everyone on the same page about what the code is doing and helps smooth the merge process.
For an open source project, having public tests helps show your concern for code quality and is an easy way for knowledgeable developers to jump into your code: how do I use your code? Well, I can always check out the tests because they better work. Across many projects, documentation is pretty rare or of poor quality. Both tests and documentation are important for your project, but while documentation fades with time, test code has a very binary usefulness.
Coming from doing most of my coding in Python and dynamic languages, this concern may not be as important in static languages like C#/Java, but I think it is still important.
Tests help verify your assumptions. Returning to my original point, failure will happen and trying to plan for is a much better solution than waiting until it happens. Dynamic typing makes it faster to write code and but pushes many errors to become runtime errors. The number of times I’ve run into date/time/datetime conversion errors in Python has definitely pushed me to test more. I can assume what code is doing all day long, until I actually test it and find that one instance when the API does something you weren’t expecting. Even assuming that you really understand dates or timezones is often incorrect.
When you run into a new problem, having a test environment setup that you can easily jump into will save time and get you fixing things quicker. The more you invest in the test environment, the easier it is to solve new types of problems and quickly diagnose problems when they arise.
How much to test, what areas to test, what type of testing to use: these questions are always up for debate. Any level of testing is good and you can probably improve. In the world of GitHub, you rarely code “alone”; someone will always read your code and making it easier for them to read and analyze is a good thing. Finally, double checking yourself is a good thing. Testing is an investment, sometimes the return may take awhile to surface, but improving your testing ability is one step to becoming a better programmer.
]]>My first thought was to do this over HTTP, with Requests for the client and Bottle for the server. I started writing some code, checked the Bottle docs for sending a streaming response, and was running with a few lines.
import bottle
import subprocess
@bottle.route('/stream')
def stream():
proc = subprocess.Popen(
'echo 1 && sleep 3 && echo 2 && sleep 3 && echo 3',
shell=True,
stdout=subprocess.PIPE,
)
while proc.poll() is None:
output = proc.stdout.readline()
yield output + "\r\n"
bottle.debug(True)
bottle.run(host='0.0.0.0', port=8000, reloader=True)
This code worked great in Chrome; console output was streamed with pauses and the connection was not dropped.
Then I started on a client in Python. Requests also supports streaming
responses, just a parameter to the standard requests.get()
, nothing too
major.
import requests
r = requests.get('http://0.0.0.0:8000', stream=True)
for line in r.iter_lines():
print line
After running the client a few times, the output wasn’t getting streamed.
Output would pop in, as if the response was fully downloading before printing.
I tried increasing the sleep amount, to see if the response was too short and
tweaking the python handling of output, trying to find some sort of implicit
stdout buffering with export PYTHONUNBUFFERED=True
. I rewrote the server in
Flask, which offers the same streaming capabilities as Bottle, but
encountered the same situation with the client not streaming the response.
After consulting with teammates, we couldn’t see the problem and moved on to
trying to stream a local SSH connection instead, which had its own host of
environment problems.
The one variable I didn’t tweak in these examples was the response size, which seems obviously when looking back now. My first mistake was not testing boundary cases: all the minimum viable tests were extremely small and not extremely different. I moved on too quickly without diving deep enough into the problem: the docs and samples were all so simple, so I assumed nothing could be wrong with the libraries I’m using.
This hackathon was all for fun, so I gave up and moved onto the next problem, but I returned the next day and dived deeper. My second mistake was trusting the docs: documentation is a great resource, but code never lies.
After looking at the function declaration for the response.iter_lines()
,
it quickly made sense: the default chunk size for the lines in the response was
512 bytes, which 1 2 3
would never get chunked into multiple pieces. I was
also not sending \r\n
, the standard HTTP chunked terminator.
ITER_CHUNK_SIZE = 512
def iter_lines(self, chunk_size=ITER_CHUNK_SIZE, decode_unicode=None):
Setting chunk_size=1
made my client immediately print output, solving all my
problems.
To make testing easier, I’ve created a Github repo to demo all this code: https://github.com/askedrelic/streaming-demo.
While debugging this situation, I remembered to try HTTPie, a cURL replacement written in Python using Requests, which handled the streaming response correctly. Looking at HTTPie’s code for consuming responses, led to me look at Request’s code and figure everything out. Definitely recommend this tool!
Lastly, always check the code and don’t be afraid to dive deep.
]]>.bashrc
.
menuvirtualenv() {
select env in `lsvirtualenv -b`; do
if [ -n "$env" ]; then
workon "$env"
fi;
break;
done;
}
alias v.menu='menuvirtualenv'
12:54:26 pcoles@peters_air:~ > v.menu
1) category-cms
2) collect
3) mrcoles
4) readmd
#? 3
(mrcoles)12:54:33 pcoles@peters_air:~/projects/mrcoles >
This method for selecting an input stood out to me for being so simple: just a numbered list. Many command line applications make input too complex, making the user think about what they want to select, making them type it in again, while many don’t even support tab complete.
Several months ago, I was on an airplane with no internet and decided to challenge myself to implement that select interface in Python. When looking at the code again, I found it was just a Bash builtin function:
select name [ in word ] ; do list ; done
The list of words following in is expanded, generating a list of items. The
set of expanded words is printed on the standard error, each preceded by
a number. If the in word is omitted, the positional parameters are printed
(see PARAMETERS below). The PS3 prompt is then displayed and a line read
from the standard input. If the line consists of a number corresponding to
one of the displayed words, then the value of name is set to that word. If
the line is empty, the words and prompt are displayed again. If EOF is read,
the command completes. Any other value read causes name to be set to null.
The line read is saved in the variable REPLY. The list is executed after
each selection until a break command is executed. The exit status of select
is the exit status of the last command executed in list, or zero if no
commands were executed.
I started hacking, got it working, then forgot it about. Now coming back to the code with an internet connection, I’ve released it on PyPI and Github.
Pyselect wraps raw_input()
, more or less:
In [1]: import pyselect
In [2]: pyselect.select(['apples', 'oranges', 'bananas'])
1) apples
2) oranges
3) bananas
#? 2
Out[2]: 'oranges'
But can also be used as a Python module, when scripting:
$ python -m pyselect $(ls)
1) LICENSE.txt
2) build/
3) dist/
4) pyselect.egg-info/
5) pyselect.py
6) pyselect.pyc
7) setup.py
8) test.py
#? 4
pyselect.egg-info/
Or in a Bash pipe:
$ ls | xargs python -m pyselect
1) LICENSE.txt
2) build/
3) dist/
4) pyselect.egg-info/
5) pyselect.py
6) pyselect.pyc
7) setup.py
8) test.py
#? 5
pyselect.py
But that’s where things kind of fall apart. Within a standard interactive Python application, stdin and stdout are simple and pyselect just works. Getting the pipe-in to work required a bit more work, hooking in/out up the user’s tty, which the pipe drops. My holy grail would be a pipe-in and pipe-out to make selecting anything much easier:
$ ls | xargs python -m pyselect | cp $0 test.txt
Or display all your git branches and jump to one. Or virtualenvs. Or directories. Double pipe redirects the pyselect output/input and doesn’t work. I’ve read up on named pipes that might be able to solve this, but I haven’t found a Python solution yet.
To jump between git branches with bash select, I used this:
function gobranch() {
select branch in $(git for-each-ref --sort=-committerdate refs/heads/ --format='%(refname)' | sed 's/refs\/heads\///g'); do
git checkout "$branch"
break;
done;
}
For now, I have some other ideas to try with selecting things:
git add --interactive
modeIdeally, pyselect could become “input for humans”, ala Requests, because
raw_input()
could always use a more friendly API.