{"version":"https://jsonfeed.org/version/1.1","title":"markentier.tech","home_page_url":"https://markentier.tech","feed_url":"https://markentier.tech/feed.json","icon":"https://markentier.tech/i/mtt.icon.png?v=20210131","author":{"name":"Christoph Grabo"},"authors":[{"name":"Christoph Grabo"}],"items":[{"id":"https://markentier.tech/posts/2023/12/cargo-upgrade-on-windows/","url":"https://markentier.tech/posts/2023/12/cargo-upgrade-on-windows/","date_published":"2023-12-31T00:00:00Z","tags":["rust","cargo","upgrade","dependencies","windows","powershell"],"content_html":"
I use Windows 11 as my main OS on my home PC, so I have to tweak some of my commands for that environment. The previously mentioned cargo upgrade is no exception.
\ncargo install cargo-edit `\n --no-default-features `\n -F upgrade,vendored-libgit2 `\n --bin cargo-upgrade\n
\nNote: Every time you see a \\
somewhere, replace it with `
to make it work in Powershell. You can also install some other shells like Bash, but making those your default is probably combined with a lot of unnecessary pain.
Since cargo
moved to a new index protocol (sparse
)1, some tools like cargo upgrade
do not really work anymore by default, since they rely on the "old" git protocol.
And since the cargo team promises to support the git indices indefinitely, we can still use them.
\nThe Powershell version looks like this:
\npowershell -Command {\n $env:CARGO_REGISTRIES_CRATES_IO_PROTOCOL="git";\n cargo fetch;\n cargo upgrade\n}\n
\nNotes:
\npwsh
instead{ … }
), no backticks needed anywhere — neat!Considering that most snippets, guides, and tutorials out there assume a Linux(-like) environment I might just go forward and always post the Windows/Powershell variant instead. I'm pretty sure I am not the only Windows+Rust developer in the universe.
\nNote: No OS love or hate here. I use all three major platforms on a regular/daily basis, I understand the trade-offs I'm making by choosing one over the other for my tasks.
\n\nSee announcement under https://blog.rust-lang.org/inside-rust/2023/01/30/cargo-sparse-protocol.html
\nIn a previous sticky I briefly put down how to get a quick'n'dirty self-signed TLS1 certificate, but also mentioned mkcert.dev, which makes it a bit better for some use cases.
\n# Windows, with scoop (it's like homebrew)\nscoop bucket add extras # skip if you have done this already\nscoop install mkcert\n
\nNon-Windows users find more ways under mkcert install.
\nmkcert -install\n
\nThis creates a CA certificate and puts it into the places to get it trusted; at least it tries to. This needs to be done only once.
\nWindows users: Run this in a powershell with administrator privileges, otherwise some steps might fail!
\n# powershell (uses ` instead of \\ for multiline commands)\nmkcert `\n -ecdsa `\n -cert-file mydomains.pem `\n -key-file mydomains-key.pem `\n markentier.test "*.markentier.test" `\n unicorn.test "*.unicorn.test" `\n rainbow.test "*.rainbow.test" `\n localhost `\n 127.0.0.1 ::1\n
\nNotes:
\n-ecdsa
option if you have troubles with this algorithm..test
is a nice top-level domain to pick, as it's reserved for—you guessed it—testing.localhost
for good measure …For some things you might need a PFX file instead of just the PEM. With the openssl
tool you can create one.
# powershell\nopenssl pkcs12 -export `\n -out mydomains.pfx `\n -inkey mydomains-key.pem `\n -in mydomains.pem\n
\nWhile mkcert
also comes with a -pkcs12
option, I usually create the regular PEM version first and only get a PFX derived from that if needed.
Not a mkcert issue, but you want to add some entries to your /etc/hosts
file if you do web development. For Windows users that's under
C:\\Windows\\System32\\drivers\\etc\\hosts\n
\nUsing some of the domain names from above:
\n### markentier.test\n127.0.0.1 markentier.test www.markentier.test web.markentier.test\n::1 markentier.test www.markentier.test web.markentier.test\n\n### unicorn.test\n127.0.0.1 unicorn.test www.unicorn.test web.unicorn.test\n::1 unicorn.test www.unicorn.test web.unicorn.test\n\n# insert more as needed\n
\nNotes:
\nThat's it.
\nHave fun with your custom testing domains. Now you can build apps expecting TLS, like PWAs (service workers), or using QUIC and gRPC protocols.
\n\nReminder, only marketing "needs" to talk about SSL, but as said in the past, you definitely only want to use the TLS versions of the protocol, nothing else. (Ideally TLS v1.3, but version 1.2 is still fine.)
\nTo create more levels of subdomain, you probably need to specify each level with one wildcard, like my.domain *.my.domain *.sub1.my.domain *.sub2.my.domain *.subsub.sub1.my.domain
… and so on …
Sometimes I need a TLS1 certificate, for some local HTTPS or other services/protocols with some level of security to play nicely.
\nMake sure you have openssl
installed on your computer. Then run:
openssl req -nodes -new -x509 \\\n -keyout server.key -out server.crt \\\n -subj '/C=EU/L=Berlin/O=MarkentierTech/CN=myservice'\n
\nNotes:
\nC
, L
, O
, CN
) to your needsSimilar name, but different use case: mkcert.org — to get a PEM file of certificates you want to trust. For example if you want to build/use apps with custom certificate trust store.
\n\nYep, I will only talk about TLS, we should really forget about SSL, because despite marketing those protocol versions shall never be used again.
\nWhile cargo already has most common commands integrated, one I miss from cargo-edit is cargo upgrade
. It's super convenient if you want to bump all dependencies at once to the latest (in)compatible version.
This is how to install just this single executable:
\ncargo install cargo-edit \\\n --no-default-features \\\n -F upgrade,vendored-libgit2 \\\n --bin cargo-upgrade\n
\nNotes:
\n--no-default-features
to avoid building all commands from a compilation perspective-F upgrade
to opt-in for the upgrade command (it's a required feature for the binary)--bin cargo-upgrade
to actually say, we want just this binary from the project, not all of them-F vendored-libgit2
: readding from the default featuresInstead of following through with Advent of Code … again, my brain thought differently: why not implement ipv4.games in Rust?
\n\nWhat is this game even? From the original creator:
\n\n\n\nI have a simple challenge for you:\n
Send requests to my web server from lots of different IP addresses.\n \n
This sounds super simple until you realize that your current device usually just has one IP address. I mean the public facing one, with version 4. IPv6 is a completely different beast, and we will completely ignore that. Maybe in an imaginary future we will tackle that, definitely no promises whatsoever.
\nI've been »playing« ipv4.games for a few weeks already, but just recently I came up with the idea I could build my own version of it, in Rust, of course, because why not. And also as usual after around two weeks of Advent of Code I get both annoyed and bored by the puzzles, as they not only require decent coding skills but also extremely sophisticated mathematical understanding of some problems. I never studied math or computer sciences, and I would have to »cheat« my way through the later day challenges, as I need several pointers to what the abstract problem is and how that one can be solved. To be honest, that removes a lot of fun for me at some point.
\nDon't get me wrong, I love the project, I even support the creator, but the puzzles become too far removed from reality that even the nice story around it doesn't help my brain to stay motivated. It's a me problem. And that's okay. I don't have to be an expert. My main driver to participate, specifically with Rust for the last four years at least, is to hone my coding skills in that language and also learn new tricks and concepts. Sometimes even a new crate from the ecosystem, though I usually try to solve most puzzles without third party libraries if possible.
\nAnyway back to the original topic, because you surely didn't want to read about my side quest of puzzle solving during end of year holiday season.1
\nSo the puzzles didn't give me the kick and I still was in the mood for programming with Rust and the IPv4 turf war was still on the radar.
\nNow even the basic principles of the service are told quickly:
\nSo far so good.
\nThe stack I've chosen is the following:
\nIn the early days everything was a single crate with a single binary. The majority of the time got sunk into figuring out the right SQL incantations for Postgres to calculate the right views for the rankings.
\nCoincidentally ChatGPT got very famous recently and I gave it a try. Oh boy was that useful! Asking a chat bot to provide some queries and also explain them was so helpful, I saved so much time. I have never tried GitHub Copilot, but the internet assures me that ChatGPT is already so much better (it's also more generic3).
\nI'm usually more sceptical when it comes to AI4, and also ChatGPT is not without controversies, but as a supplementary tool for programming I believe it's quite valuable. Side note: I also do not believe that such AI (chat) bots will make our profession obsolete anytime soon.
\nFirst: someone still needs to build and program the tools to begin with, they will also require constant improvements and maintenance.
\nSecond: there is still a vast amount of existing software out there in desparate need of our attention. That should occupy us for a long time before AI steps in.
\nThird: they will not be able to spit out complete start up businesses or full programs. This is already a super difficult task for human beings. If you have a super great business idea you still need to encode that into a precise enough language without any ambiguity left, so a machine can stimulate its neural network and machine learned algorithms to solve for that. Maybe one day we will get there, but currently the AIs still struggle even with some simple tasks at times. ChatGPT did spit out syntactically wrong queries, which were not a biggy for me though.
\nIt reminds me of OCR5 and voice recognition, both technologies have been around now forever (a century at least for text), and yet the error rates are sometimes still too high to be used reliably for all potential use cases. I'd say every time there is a human involved in the process (as the user/actor) such systems need to be extremly sophisticated to support the eight billion beings on this planet. Good luck to make that fool proof.
\nWhere was I again? Ah, yes, computer aided computer dabbling.
\nLong story short, I saved valuable time compared to searching the interwebz with Google and StackOverflow. I'm not an SQL pro, and my last active usage was in context of MySQL and is also a few years ago, so I forgot even some basics, let alone dialect specific intricacies (Postgres is closer to the SQL spec though, but who knows specs by heart anyway).
\nI also quickly switched from sqlx to diesel, since the former was somehow a bit too painful for me to use. While not an ORM and in my case also not really necessary I struggled with it mostly because of some super strict macro voodoo, which I couldn't really endure for long. My preferred guidance system called rust-analyzer was yelling constantly with blocks of red underlined code — and the messages were not always clear to me, why it was yelling so much. Perhaps I used it wrongly and/or inefficiently, but I then made the call to switch the tools and diesel was actually way more pleasant to use. I might have lost some super duper fancy compile time checks here and there, but granted, my use case doesn't demand too much in that area anyway.
\nMy verdict is also: while macros can be very helpful at times, please also don't overdo it. And I see a trend that more and more crates are either refactored or alternatives created to avoid heavy macro usage if possible. I appreciate that, and so does rust analyzer, which still cannot deal with all invocation patterns.
\nThe serverless Postgres instance provided by neon.tech proved to be really straight forward, but again, I don't have special needs here, a stock database does the job for me. My Windows host needed a nudge with the openssl version, but that was not a big deal either; I'll come to that a bit later again.
\nAfter I got the foundation of the application working I did several things afterwards.
\nInstead of a single flat crate I transformed the project into a cargo workspace with several member crates. One (future) reason for that was that the migration tool doesn't work nicely with the async setup for the main app, so splitting out some responsibilities made sense. Furthermore it's always an interesting challenge to organize the logic into some distinct pieces. Of course, I went totally over board with that and now I have crates for a few constants and types and environment variable readings.
\nSome separation absolutely do make sense though: the DB6 schema is a diesel managed thing and the business logic code doesn't need to know all the internals of that, it's also partially auto-generated code, so having its own place is preferred. Last but not least the assets for the frontend (HTML, CSS, and JS files) which are compiled into the binary/library is a crate on its owned, too.
\nSpeaking of assets and frontend …
\nLike the original I also chose to deliver static HTML (aside from the tiny dynamically rendered claim page), some vanilla JavaScript, sprinkled with some colourful CSS. No React, or Vue, or Svelte, or whatever the next hot shit on the frontend web framework front is. I could have. I maybe should have, since the objective was to over-engineer the solution. But somehow I felt that I did not want to invest too much on that particular part. I have been burnt to often by npm and webpack and rollup and grunt and gulp and parcel and even swc and JavaScript based tooling in general, I just could not do it. If I wanted to suffer I wanted to take control of where I want to suffer, and frontend was not it. So I shamelessly copy-pasted most of the code from the ipv4.games site and adapted it to my needs.
\nI believe it's still very efficient and I also cannot imagine that a frontend library/framework can really do a much better job. The fetching and rendering are already pleasantly fast (with a decent internet connection), and the point of optimization lies elsewhere anyway.
\nI did some unrolling and hardcoding for the IP block overview, which improves the rendering, as the layout doesn't shift that much, only data in the boxes gets populated. I should probably do similar changes to the top and recent lists as well. On the other hand my site doesn't really need to be super SEO optimized that much. A regular user should have a decent visit experience and that's okay for now.
\nLet's get back to the backend again, it's more fun. At least for me.
\nFurthermore let's pay our attention to the core service, which does come in two pieces. The migration stuff is not super interesting here, so I'll leave it out. (You can also completely rely on diesel's CLI to manage migrations anyway.)
\nSince axum builds on hyper, and hyper runs in an async environment like tokio, everything is pretty asynchronuous. Whatever that means, right? What I have noticed over the last few months in /r/rust on reddit is that many people still use rocket as their framework of choice. There is nothing wrong with that, but what's interesting is that the current stable release (v0.4) is not async, but instead relies on multi-threading. Version 0.5, currently available as an RC (release candidate), will change that, but development seems a bit slow and users have been waiting for that to come to live for a while; the RC is from May and the year is almost over.
\nNow, multi-threading can work and work well enough, but unless your web services does heavy number crunching on the CPU it will mostly wait for some I/O[^] interaction like file system or network, which also includes communication with the database in most cases. The beauty of asynchronicity, especially on a single thread is that during that wait time (so until the operating system tells you that your stuff you're waiting for is ready) you can already do other things concurrently. Like serving requests for other users for example. With threads that would usually involve context switches, and without diving deeper here, let's just remember that those context switches are more costly than staying in your current thread. Your computer will appreciate you for that. Or not, who knows.
\nThat brings me also to a tiny nitpick with the widely used runtime tokio, which promotes in its default mode still a multi-threaded setup, unless you explicitly tell it not to. And I guess this is a trade-off decision here, but I do wonder if the majority of users will really need the multi-threaded setup at all.
\nDon't get me wrong, a single thread alone is not always great, even Node.js, the JavaScript runtime for backends, is actually not really single threaded either, it has thread(s) for off-loading blocking I/O for example. And in my particular case I also have adjacents threads: the deadpool library spawns threads for managing the database connections, and I also implemented a dedicated thread for background work.
\nThe potential performance impact is that on fly.io I picked the free tier machines which come with a "shared CPU", which is probably also just a slice of a real physical CPU (to be honest I don't know how their compute scheduling works internally). In an ideal scenario you would spawn only as many threads as CPU cores, so context switches per core would be low to non-existent, but that's usually a pipe dream anyway. In our hobby project we're definitely in the real of unnecessary micro optimizations.
\nThe other famous (now async) framework actix lost me a few years ago, and until beginning of this year I also dabbled with tide, which might have lost the popularity contest due to low adoption of the alternative async runtime async-std. As much as I like(d) tide and async-std I gave into the better ecosystem and community support around tokio and hyper. warp would have been another option (also running on tokio/hyper), but quite frankly I always found the way of composing the routes very odd and cumbersome.
\naxum is relatively new on its own, but thanks to the tokio, hyper, and tower ecosystem, it does not start from scratch. It's a simple framework without too many frills and batteries, similar to Sinatra in Ruby or express/hapi/koa in Node.js land. A full-blown Ruby on Rails clone is not present … yet. But question is also if that's even desirable. Within the last 12 years of my professional career I've seen a steady decline for the big monolithic all-batteries-included highly-opinionated monster towards more smaller, concise, single purpose services, where big frameworks are not necessarily shining or needed.
\nThe router and handlers (the functions doing that endpoint work) are nothing out of the ordinary, the API design is following the original implementation, the payload shape slightly adjusted, some JSON here, some query paramenters there, some DB pool state over here again.
\nSince the endpoints don't really do a lot—asking the database for stuff and transforming that thanks to serde to JSON responses—let's talk about the data model for a bit. I'm pretty sure if I twist that Postgres stuff a bit further, the endpoint wouldn't have to do anything but stream the bits and bytes as-is from end to end. (I believe that exists already and is called PostgREST.)
\nThe core data model is a single table. Yep, just one table. That table holds information about who claimed which IP and when. So in the worst case the maximum size of the table is the amount of all possible IP addresses: 4,294,967,294 (2 to the power of 32). Now not every IP is technically usable, at least not without some spoofing. And there are ranges which are reserved for special purposes, some are reserved for the future, some serve broadcasting and multicasting needs, some blocks are in private or governmental hand, some are just private by nature (local/internal networking), … long story short, huge ranges and blocks of IPs are just not assignable easily or at all. Even then, we have seen tables which reached the maximum 232 value for their ID column and people realized that we should have switched to 64 bit integers for such identifiers a long time ago, meaning, our databases are able to handle such volumes of data.
\nThe other "tables" are actually materialized views, a very nice and handy feature of Postgres, which are like normal views, but persisting their results once calculated. Views are saved queries which you can also filter further if needed.
\nI use materialized views since that is a nice trade-off between query performance, caching, and compute time. The information of those views does not need to be realtime. And the mentioned background worker thread is running in an interval to refresh the views, currently set to every 15 seconds. This approach of database side caching is pretty neat, and will scale for the time being; I don't have to think about any CDN7 and caching and purging strategies. Just every quarter minute some data gets updated, that's it. Besides using materialized views, that's probably even the opposite of over-engineering, since it's more of a very pragmatic approach instead.
\nThe building of the queries as mentioned in the beginning took some time, because crunching out the right results as needed and expected where not that simple. Some of the queries use multiple CTEs (Common Table Expressions, those things with the WITH
keyword), an alternative to deeply nested sub-queries. I don't really know if they are better or worse, but in cases where I used the same CTE more than once I believe they should be more efficient (since reusable in same context). I could run EXPLAINs and try to decipher the query planner results, but I was satisfied enough with the bare query timings, and then saved as materialized views those timings were even better, so all good in my book for now. If that thing falls over one day, no real harm will be done, it's a toy project, no revenue impact, only hurt feelings, maybe.
I already mentioned that openssl under Windows issue a bit, it was just a slightly outdated version which didn't supported the latest and greatest of what TLS8 (or for the marketingese speaking people: SSL9) can offer, so the postgres client was not happy when I tried to connect securely to my neon.tech database. A bunch of vcpkg updates and recompilations later that was no issue. But it made me think: do I actually want to rely on openssl directly? It's a tool written in C, not Rust, and while probably has seen better vetting than any new alternative I wanted to somehow stick closer to Rust if that was possible. And history has shown that openssl had bugs.
\nThe first step was actually switching to native-tls, which means: use the TLS stack of the platform. For Windows that is SChannel, for macOS the Security.framework, and only for all other platforms it would be openssl.
\nA day later I then moved on to a rustls stack, so plain Rust, no 3rd party bindings to some non-Rusty thing. And let's be real, the rustls logo is cool. (A slight bitter pill: ring, the underlying crypto library is a mix of Rust, C, and assembly, so not pure under the covers; at least it doesn't come with 3rd party dependencies, which have to be present in the system somehow.)
\nThe integration was a bit tedious but thanks to a very recent update to diesel-async I could customize the database pool setup to utilize my preferred crates. The only exception is the cti_migrate binary, which uses diesel's EmbeddedMigrations offering, which in turn is tied to diesel's stock implementation for Postgres connections. Since I don't have to migrate all day long I didn't bother to jump down that rabbit hole. That's something for future me to investigate. Or simply stick to the diesel CLI.
\nSince fly.io offers not only HTTP/1.1, but also HTTP/2 I fiddled around with that setup a bit. The documentation around that is a bit scarce and the community forum was also more a scavenger hunt for snippets of details than definite answers. The probably best configuration would be to let fly's edge terminate TLS for me and pass through the bare TCP connection down to my service. But there is a downside: the client IP is not automatically handed down with that, yet that's actually a very important piece of information for me, that's the whole purpose of the game!
\nThe extended variation to that would be to enable something called proxy_proto
, which means the edge talks the PROXY protocol as defined by haproxy, which would pass some valuable headers. Now resources, let alone crates for that protocol in Rust are almost non-existent, there is one crate which seems to understand the header format, but I think I would have to implement a lot myself, especially if I want my service to basically act as both a proxy and a backend service in one package.
The only viable solution would be to put a real proxy like haproxy in the middle, but somehow that sounds suboptimal to me. Usually each component in the chain of a request will add some latency. I did ran an experiment in a second fly app, and even got something working, but I have to trial it in my actual game setup and see if that even provides any benefits. For now I let the edge still just talk HTTP to the backend, all version 2 already (thanks to h2c
, meaning cleartext HTTP/2, which is basically only supported outside of browsers by the way).
What about HTTP/3? That thing is still in its infancy and needs more time to grow, I think only Cloudflare fully supports it, AWS CloudFront also just got it recently. The Rust ecosystem is also not there yet, as far as I know no web framework has built-in support for that new protocol, which is vastly different from the previous HTTP implementations. It will take time until version 3 can replace its older siblings.
\nLastly I tried something new for me: since I already had two binaries with shared logic (though I also only use one of them in production), what if I don't link the shared code statically, but instead dynamically?
\nIf you're on a Windows machine you might have heard of those DLL files, and on a Linux machine those pesky files with .so extensions might have brought you pain, because probably something expected a different version of glibc (something something libc.so.6 not found), yeah that's all about shared libraries. Sounds fun already, doesn't it?
\nSpecifically software package and Linux distribution maintainers tout them as the holy grail, since they make patching security holes for all dependent programs a breeze. Unless version mismatches makes your life miserable instead. The industry went more into the opposite direction (fully packaged and isolated software bundle solutions like Flatpak and Snap), I guess not everyone agreed with the maintenance message.
\nYet in my case it was a simple "I control both ends of the equation" problem, which is quite easy to solve. First I reached for a crate called libloading, which is for example also used for dynamic linking in the bevy game engine. But the weird thing about it is that the information about a dynamic/shared depencency is not visible from the outside, meaning tools like dumpbin (Windows) or ldd (Linux) won't tell you that this binary expects more dependencies than what's stated, since the loading mechanism is part of the actual program code.
\nThe "easier" solution is to use the same dynamic linking mechanics as everything else would do: declare the external function stub, tell the linker about the required library for that declaration and where to find it. Et voilà, if done right the executable now also explicitly tells the rest of the world that it requires something else from the system.
\nThe admittely unexciting output might look like the following:
\n$ /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --list /app/bin/cti_server\n linux-vdso.so.1 (0x00007ffd94b74000)\n libcti_core.so => /lib/libcti_core.so (0x00007fefff40b000)\n libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fefff3e9000)\n … snip …\n
\nThe second line of the list (libcti_core.so
) now officially points to a shared library which was just compiled a few minutes ago and put in place there to be found by our binary. If you want to learn more about how binaries get called and executed and also find their shared dependencies, I highly recommend to watch Ryan's talk about Rust Before Main.
Just a quick word on that part: it's fast, and it's small, both in size and resource usage.
\nIn idle times the CPU utilization looks more like some ground noise:
\n\nThat one spike there is a deployment, where the CPU went up to roughly 5.5 %. Yes, that's not a typo, less than six percent of that shared CPU.
\nOf course, there is a way to drive that up, I did some stress testing, so 20 to 60 % I managed to achieve myself. I also had some visitors rushing in with their tens of thousands of different IP addresses at once. The response times peaked, some requests had to wait for a few seconds, because I only have allocated 3 instances across Europe and the US, but as far as I can tell, nothing broke, neither web app nor database; everything and everyone got served. Excellent job of Rust, fly.io, and neon.tech — thank you so much!
\nMemory-wise the service hovers around 32 MB on a 256 MB instance. Since there is no real state during any call that number stays relatively flat, no matter what you do. No real state, no leaks, nothing, nada. This is amazing.
\nThe common response time for a user from Germany with a decent DSL connection should result in 25 to 50 ms per call/endpoint, sometimes a fresh/cold call might result in 200 to 300 ms for the index page in the worst case. (Other locations might see higher latencies, since the main region is Frankfurt, Germany — also for the database for that matter.)
\n\nWith some parallel calls this is done in 3 batches taking not more than 120 ms to fetch, then a bit for the processing and rendering which also shouldn't take too long. In less than a second the business is done.
\nThe static assets also have some cache-control headers, so repeated calls don't cause performance penalties aside from the uncached API calls.
\nI wonder if AWS can even beat that, but the last time I tested CloudFront there were some inherent latencies due to their infrastructure design, and that was for Lambda@Edge, where you couldn't go below 20 ms at all. I believe the performance is even worse for apps served directly from a region (like via API Gateway for example).
\nWhat I'm saying is that I could for example build a blog serving application, where everything is delivered from the app (static assets compiled in, content pages computed on the fly), no CDN and no caching, and it would still perform equally well like my static Netlify site.
\nWhile I praise them for their great performance, there are also some pretty annoying things.
\nA friend of mine told me that he couldn't deploy a Django app. For me the trouble is more around deployments when you run more than one instance. Weirdly (thanks to a tip from the community forum) if I scale down to 1 first, deploy the new version, and then scale up to 3 again, it works quite okay.
\nEven then sometimes their infrastructure has tiny hiccups. An instance might hang or the scheduling is heavily delayed. I wish for fly to succeed, but they have to invest in the stability and reliability of their platform. I can deal with such shenanigans, I'm a tinkerer, I have no monetary stakes in the game here, but for businesses such things are just a no-go. Generally I believe they go in the right direction, the firecracker based runtime environment seems to work and perform well, and I can just hope other cloud/edge infrastructure providers will keep an eye on that and adopt same or similar patterns.
\nAs of now I consider the project feature complete and I also learned quite a lot of different thinks across the whole stack in Rust. There are potential future excursions I could take, like the mentioned end-to-end HTTP/2 bridge, maybe exploring HTTP/3 with a different provider/hoster, CDN and caching (which I have enough of at work to be honest), fancy frontend framework, maybe a Rust/Wasm based one, twists on the IP capturing scheme, daily challenges maybe, a version for IPv6 (though very unlikely as this number space is too huge and requires a different approach), more physical separation of the web service and the background worker (which could be kind of a cron job, AWS Lambda might come to mind here). If I think long enough, I might have more ideas.
\nI already spent quite so much time the last two weeks on it, I probably need some distance from it for a while though.
\nIf you want to dive into the codebase and check out how I did some things and sometimes even why, go over to GitHub and have fun:
\n⇛ https://github.com/asaaki/capture-the-ip
\nFeel free to open an issue with questions and suggestions, and don't shy away from opening a pull request if you know something I should improve or change.
\nUse it as an inspiration for your next project if you like.
\nAnd dont' forget to give ipv4.quest a try!
\n🙇10
\n\nOr did you? 🤔
\nObject-Relational Mapper, funny thing to translate between your objects (in Rust: structs) and the data stored in the database.
\nThis AI4 driven chat bot is not focused on programming at all, but it turns out that the training material must have been so vast, that it does an excellent job in providing help in such particular domains.
\nArtificial intelligence, the opposite of natural intelligence I suppose, though for the latter we don't really have proof, do we?
\nOptical Character Recognition, also known as text recognition
\nShort for "database", but coul also be something like digital blob maybe; what are databases even really? Aren't they just a fancy abstraction layer on top of the file system, with even fancier access capabilities in the shape of a query language more powerful than your common shell can provide?
\nInput/Output, describes the communication between systems within or across computers.
\nContent Delivery Network; just a bunch of servers across the globe, caching the content close to the users
\nTransport Layer Security, the successor to SSL9. That stuff encrypts your internet traffic, so nobody in the middle can read what you communicate with the server, or tamper with it.
\nSecure Sockets Layer, the predecessor to TLS8. Old version, don't use, broken and thus dangerous. Sadly the term SSL
is still widely used, because it really became more of a recognizable brand and marketing term, even when people actually mean TLS
.
I have just written that whole article down in one go without any review so far. I want it to get out, rather sooner than later. Thus I skipped even the proof reading I usually do. If you find any mistakes (before I do), you can keep them as a souvenir. Or ping me on mastodon and complain about it: https://mastodon.social/@asaaki
\nThe average life span of a human is around 80 years. Today marks my halftime. I have no freaking clue what to do with that second half. No plot twist here.
\n\nTo be honest, before I hit 20 I didn't even expect to ever get twice as old then. So technically I'm dead already according to my own prediction. Well, screw you past me, I outlived your expections.
\nSo I'm an adult now. I should know now how this whole life thing works, right? Wrong again.
\nMaybe I get a better idea of it in my second half of life. Maybe never. And it probably doesn't matter anyway.
\nTime for some good old midlife crisis … 1
\n\nThe phenomenon is described as a psychological crisis brought about by events that highlight a person's growing age, inevitable mortality, and possibly lack of accomplishments in life. This may produce feelings of intense depression, remorse, and high levels of anxiety, or the desire to achieve youthfulness or make drastic changes to their current lifestyle or feel the wish to change past decisions and events. — https://en.wikipedia.org/wiki/Midlife_crisis
\nDeveloping on Windows & Linux? Using WSL2 but repos are on NTFS disk? Having slow build times? Then compile elsewhere!
\n\nI have been using Windows 10 as my main operating system for quite some time now (~ 3 years). And generally I'm also quite happy with it, even for software development purposes which do not target Windows itself as a platform.
\nFor example I play with and build Rust programs which usually need to run in a non-Windows environment like Linux. The folks in Redmond made it a pretty nice experience with WSL2 (Windows Subsystem for Linux, version 2). But one huge issue remains: speed across file system/VM boundaries.
\nMore specifically: if you have a project on an NTFS partition and use it within a Linux guest via WSL, then you might notice very slow operations when it comes to file handling.
\nI wrote about this a while ago for git command related issues. The trick there was to use the correct git from either side (WSL/Linux vs Windows), but that won't help you in all cases.
\nIf you already use your Linux only tools you still get impacted by the file system, … and Microsoft still hasn't made progress on this issue. I guess we will have to wait for a WSL3 probably.
\nUsually the recommendation is to host your files in a Linux file system within WSL2, but as I mentioned in my git post above I don't really want that. One reason is that using such projects directly under Windows becomes cumbersome again (try find a reliable and affordable tool to mount Linux file systems in Windows, none of the existing solutions really increase my confidence so far).
\nRecently I participated in the Advent of Code (AoC), well at least until day 15, after that the puzzles became a lot of work and I didn't feel like spending further time on them considering I also wanted a bit of a break from work itself.
\nAnyway I learned still quite a lot:
\nThe last part was actually quite annoying, since the compilation times were significantly higher in the WSL2 environment than in the host. And after a while I got reminded of the git issue I had.
\nSince I was compiling a lot I definitely wanted to address the issue and I came up with a quite decent solution.
\nWhat if we compile on a Linux file system and therefore just shuffle around files?*
\n*) smartly!
\nThe idea is the following:
\nThings to skip on the way are:
\ntarget
folder - we neither want the data from either location copied into the other.git
folder - unless the build depends on git data, we can skip it; in the future it might be also possible to just set some environment variables if the build script and/or dependency can work with itWhile cp
is okay as a tool I actually went for rsync
, as it provides a nicer interface when it comes to ignoring paths. And as far as I know it's also smarter about if and what to copy over.
Bonus level: using a RAM disk as a temporary build location. But this is definitely optional.
\nFirst as a teaser the compilation time comparison:
\n# in WSL2, clean builds (cargo clean before a cargo build)\n\n# - from NTFS location:\nFinished dev [unoptimized + debuginfo] target(s) in 1m 30s\n\n# - from "Linux" (ext4) location:\nFinished dev [unoptimized + debuginfo] target(s) in 21.39s\n# ~ 4 times faster\n
\nAnd the results for release builds (opt-level 3, lto, codegen-units 1):
\n# - from NTFS location:\nFinished release [optimized] target(s) in 2m 00s\n\n# - from "Linux" (ext4) location:\nFinished release [optimized] target(s) in 48.27s\n# ~ 2.5 times faster\n
\nI consider that a huge boost. Especially on warmed dev builds that allows for very quick compile+run tests if one desires. A changed println!
can recompile in roughly one and a half seconds instead of twenty for example.
First designate a location which is in the realm of your Linux environment, your home folder might be a good start.\nSince I use it for temporary builds, I always have a user tmp
folder, so the full path would be /home/<username>/tmp
. Under that you might then prefer to have your project folder name repeated to easily find the stuff again.
Second you need to have rsync
installed. On a Debian or Ubuntu system the following command should help:
sudo apt install rsync\n
\nAnd if you like a tiny bit of automation I still recommend make
:
sudo apt install make\n
\nBut you can adapt the following snippets for your build environment as well (maybe you prefer just or cargo-make for example).
\n# very minimal setup\n\nSOURCE_DIR = $(PWD)\n# `notdir` returns the part after the last `/`\n# so if the source was "/some/nested/project", only "project" remains\nBUILD_DIR = ~/tmp/$(notdir $(SOURCE_DIR))\n\nbuild.wsl: wsl.sync\n\tcd $(BUILD_DIR) && cargo build\n\nrun.wsl: wsl.sync\n\tcd $(BUILD_DIR) && cargo run\n\nwsl.sync:\n\tmkdir -p $(BUILD_DIR)\n\trsync -av $(SOURCE_DIR)/ $(BUILD_DIR)/ --exclude .git --exclude target\n
\nYes, the trailing slashes in the rsync
command are necessary.
When triggering a make run.wsl
the project gets copied, compiled, and executed, all in a quick breeze.
Copying the artefacts back to the origin location and adjustments for using different profiles is left as an exercise for the reader.
\nA slightly more elaborate example can be seen in my AoC repo:\nhttps://github.com/asaaki/advent-of-code-2021/blob/main/Makefile
\nThere you also see how I dealt with debug and release profiles and other shenanigans.
\nThere's a lot more things you can do to improve compile times, but this post was only about showing the easy win in a WSL build environment.
\nIt is a nice tool, but once I had it installed in both environments and somehow they loved to interfere with each other, especially within WSL I never really got a long-term reliable setup running.
\nI seem to run into a similar issue as reported here:\nhttps://github.com/mozilla/sccache/issues/1067
\nIt's use case is also more about caching and sharing compile artefacts across projects, but you still have to deal with your project files in the "twilight zone" (the NTFS location in WSL). And if you have peeked into the target folder you might have seen a lot of files which cargo/rustc needs and have nothing to do with sccache at all.
\nIt's great for creating shippable artefacts, but not so much for quick development cycles. Since I do have WSL and can do further cross compilation within it, I don't depend on a dockerized development environment all the time anymore.
\nAlso creating the build containers alone can take a lot of time already. A nice exercise every once in a while, but not my everyday cake I want.
\n"},{"id":"https://markentier.tech/posts/2021/12/blank/","url":"https://markentier.tech/posts/2021/12/blank/","date_published":"2021-12-31T00:00:00Z","summary":"This page is intentionally left blank.","content_html":"This page is intentionally left blank.
\n\n2021 is finally over.
"},{"id":"https://markentier.tech/posts/2021/11/multiarch-container-images-for-docker-and-kubernetes/","url":"https://markentier.tech/posts/2021/11/multiarch-container-images-for-docker-and-kubernetes/","date_published":"2021-11-27T00:00:00Z","banner_image":"https://markentier.tech/posts/2021/11/multiarch-container-images-for-docker-and-kubernetes/cover.png","summary":"You run clusters on Raspberry Pis and Intel NUCs? You love ARM, but also need to provide images for your PC friends? Fret no more, learn about manifest lists and how BuildKit makes your life easier.","tags":["docker","kubernetes","container","image","multi-arch","multi-platform","platform","architecture","amd64","arm64","processor","dockerfile","dockerfile:1-labs"],"content_html":"You run clusters on Raspberry Pis and Intel NUCs? You love ARM, but also need to provide images for your PC friends? Fret no more, learn about manifest lists and how BuildKit makes your life easier.
\n\nMost of us will probably use Docker and Kubernetes with one platform only.\nThe predominant platform being "linux/amd64"
.1 But due to the rise of ARM based devices and cloud services, "linux/arm64"
is seing some traction as well. And if you're into embedded/IoT/microcontrollers you might come across "linux/arm/v7"
or "linux/arm/v6"
.
Depending on your environment you either build your Docker images already on the target platform or utilize cross building/compilation. Commonly you then push your platform specific images with distinct image names or tags.
\nYet Docker also supports creating multi-platform manifests, so you can consolidate some efforts, mainly around handling different image names and tags for your diverse runtime needs.2
\nWait, what? Why do you say "manifests" and not "images"?
\nWell spotted, dear reader.
\nWithout going into much further detail, the docker registry doesn't only store your images, but also some metadata around it. In reality an image is composed of different layers (at least one) and since such layers can be used by many different images, they are also separate entities. The manifest is the file that stores all the information around the image and layers, so when you run docker pull
the program knows what to fetch. That's also why you see the multiple lines during the download phase, unless you're "lucky" and fetch a docker image consisting only of a single layer.3
So, what is a multi-arch docker image?
\nFirst, I'll use multi-arch over multi-platform, as we will only focus on "linux" platforms. So both terms are used interchangeably by many people, for the better or worse. I believe it's okay, because you might be concerned about different processor architectures only, less so about running container workloads over different operating system families. (I guess game developers might be a good exception from this rule though.)
\nSecond, the answer is quite simple. One of the supported types of the docker registry is manifest lists. And the content is also very boring, basically only a list of actual manifest including which platform they target. Yep, that's it.
\n# using buildx's inspection, as it provides the information from the registry\ndocker buildx imagetools inspect --raw nginx:alpine | jq\n
\nThe buildx
command comes from the BuildKit plugin, which should be included in recent versions of the docker engine/desktop application.
{\n "manifests": [\n {\n "digest": "sha256:f51b557cbb5e8dfd8c5e416ae74b58fe823efe52d9f9fed3f229521844a509e2",\n "mediaType": "application/vnd.docker.distribution.manifest.v2+json",\n "platform": {\n "architecture": "amd64",\n "os": "linux"\n },\n "size": 1568\n },\n {\n "digest": "sha256:02216f2fc478aa25afebef2e9f39507cc04445ce092ed96adb90983006bf5286",\n "mediaType": "application/vnd.docker.distribution.manifest.v2+json",\n "platform": {\n "architecture": "arm",\n "os": "linux",\n "variant": "v6"\n },\n "size": 1568\n },\n // cut for brevity; more platforms in original output\n ],\n "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",\n "schemaVersion": 2\n}\n
\nA single manifest looks like this:
\ndocker buildx imagetools inspect --raw nginx:alpine@sha256:f51b557cbb5e8dfd8c5e416ae74b58fe823efe52d9f9fed3f229521844a509e2 | jq\n
\n{\n "schemaVersion": 2,\n "mediaType": "application/vnd.docker.distribution.manifest.v2+json",\n "config": {\n "mediaType": "application/vnd.docker.container.image.v1+json",\n "size": 8892,\n "digest": "sha256:b46db85084b80a87b94cc930a74105b74763d0175e14f5913ea5b07c312870f8"\n },\n "layers": [\n {\n "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",\n "size": 2822981,\n "digest": "sha256:97518928ae5f3d52d4164b314a7e73654eb686ecd8aafa0b79acd980773a740d"\n },\n {\n "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",\n "size": 7251533,\n "digest": "sha256:a4e1564120377c57f6c7d13778f0b12977f485196ea2785ab2a71352cd7dd95d"\n },\n // cut for brevity, original output has more layers\n ]\n}\n
\nIt's also documented and explained on Docker's site:\nhttps://docs.docker.com/registry/spec/manifest-v2-2/
\nBy the way if you're interested what's in the container image config (since the media type is JSON, so we might wanna take a peek), you can run the following command and get a glimpse:
\ndocker image inspect nginx:alpine --format '{{ json . }}' | jq\n
\nI leave the exercise for you, this post has too much JSON already. Also the output is what you would expect: it contains all the meta information and default values specified during the build of the image (environment variables, entrypoint and command, labels, …). More details under https://github.com/moby/moby/blob/master/image/spec/v1.2.md.
\nSince we learned, that multi-arch images are not really magic, let's build our own. There are two ways of doing it. Also both require a registry to push the data to. Other articles out there fall back to the public Docker Hub, but you can also do everything on your machine as shown here.
\nSince the local docker environment doesn't really play nicely with multi-arch images, a registry is needed for storage. You will also notice when you pull an image, you can only have one platform version at a time with the same image tag. The last pulled one keeps the tag, all previous ones become untagged (unless you re-tag them before pulling another one, of course). In practise only a minor inconvenience though.
\nFor both approaches we need a local registry.
\ndocker run \\\n -d -p 5000:5000 \\\n -v registry_data:/var/lib/registry \\\n --restart=always \\\n --name registry \\\n registry:2\n
\nYou can leave it running in the background.
\nIf you need to clean up, don't forget to stop the container and remove both container and volume.
\ndocker stop registry\ndocker rm registry\ndocker volume rm registry_data\n
\nIn a folder create the following file:
\n# syntax=docker/dockerfile:1-labs\nFROM alpine:3.15\n\nCOPY <<-"SCRIPT" /info.sh\n\t\t#!/bin/sh\n\t\techo "I am running under machine type (architecture):"\n\t\tuname -m\nSCRIPT\nRUN chmod +x /info.sh\n\nCMD ["/info.sh"]\n
\nMore information about # syntax=docker/dockerfile:1-labs
and also what's currently in the labs channel can be found at https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md. Here it's used for the multiline heredoc on the COPY
command.
Note: the script is indented with tabs (\\t
) not spaces. The latter won't work, the heredoc syntax only supports tab stripping.
For this you need a recent docker version, ideally 20.10 or better for native BuildKit support without fiddling around with configs and settings and env vars.
\nSo all this application will do is to print a bit of text. uname -m
returns the machine name (which is usually the same as processor (-p
) and hardware name (-i
)). For amd64 based images that should be x86_64
, and for the arm64 version a value of aarch64
instead. You could also change it to uname -a
if you want all the information availabe, like which Linux kernel it is running on.
Following steps will create two individual images (and manifests) and then combine them into a new one.
\n# run in the folder of Dockerfile and info.sh\n\n# first let's build the multiple architectures and tag them appropriately:\ndocker build --platform linux/amd64 --tag localhost:5000/myapp:amd64 .\ndocker build --platform linux/arm64 --tag localhost:5000/myapp:arm64 .\n\n# second push the images\ndocker push localhost:5000/myapp:amd64\ndocker push localhost:5000/myapp:arm64\n\n# now let's combine the manifests into a single list and push it\ndocker manifest create localhost:5000/myapp:latest \\\n --amend localhost:5000/myapp:amd64 \\\n --amend localhost:5000/myapp:arm64 \\\n --insecure\ndocker manifest push localhost:5000/myapp\n\n# check the image manifest list from the registry:\ndocker buildx imagetools inspect localhost:5000/myapp\n
\nAs you can see, this is very tedious and includes many steps.\nOn the plus side: you can parallelise that when using a CI/CD pipeline. And depending on the architecture support of your environment or provider you even need to do such fanout anyway.
\n(The reason for the --insecure
flag is explained in the following section.)
Note: BuildKit only supports Linux as a target.
\nFirst let's prepare a dedicated builder instance for this. You might have already a default builder (check with docker buildx ls
), but it uses the docker
driver and that doesn't work particularly well for creating multi-platform images; if you try you will see the following message:
\n\nerror: multiple platforms feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")
\n
That's exactly what we will do (just with some more arguments):
\ndocker buildx create --name mybuilder --driver-opt network=host --use\ndocker buildx inspect --bootstrap\n
\nYou could also add --driver docker-container
, but BuildKit defaults to this when creating a new builder. The network=host
option is there to allow pushing from the build container into the registry directly, otherwise that part would fail.
Note: You will need to export your images in some way anyway, otherwise the image artefacts stay in the builder container, which is not very useful. So pushing to a registry is recommended.
\nThe second step starts up the build container and displays information about the builder. If you remove the bootstrap flag, the container will be created the first time you need it.
\nThe inspect subcommand should return something like this:
\nName: mybuilder\nDriver: docker-container\n\nNodes:\nName: mybuilder0\nEndpoint: unix:///var/run/docker.sock\nStatus: running\nPlatforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6\n
\nImportant is the "Platforms" line. For our experiment it must include linux/amd64
and linux/arm64
.
In case you can build all your desired architectures on a single machine, the following steps are needed:
\n# run in the folder of Dockerfile and info.sh\n\n# create all the images and push all the manifests\ndocker buildx build \\\n --platform linux/amd64,linux/arm64 \\\n --tag localhost:5000/myappx:latest \\\n --output=type=registry,registry.insecure=true .\n\n# check the image manifest list from the registry:\ndocker buildx imagetools inspect localhost:5000/myappx\n
\nFor a localhost registry you can also replace --output=type=registry,registry.insecure=true
with the shorthand --push
; since I play with k3d
4 which also has an option to create registries, but with different host names, so the longer option is needed, as I don't want to switch to TLS and authentication for local testing.
There are alternative output options, but most of them are around storing artefacts locally.
\nYou can keep the builder around, but if you want to clean up, run the following:
\ndocker buildx rm mybuilder\n
\nSince you managed to create the images for the different processor architectures you can also run the containers.
\ndocker run --platform linux/amd64 --rm localhost:5000/myappx\ndocker run --platform linux/arm64 --rm localhost:5000/myappx\n
\nIf everything works well you should see two different outputs, each telling you under which architecture they run.
\nNow you know that multi-arch docker images are no magic. They are just a bundle of manifests, each pointing to an image specifically created for a platform.
\nUnless manually specified your docker engine or kubernetes cluster will pick the platform suited for its environment.
\nHave fun building a more diverse world!
\n\nThe "platform" is a tuple of operating system (OS) and processor architecture values. "linux" is the major OS, but you could also build "windows" images if you work with Docker Desktop under Window. For architectures you have a much wider variety of options, "amd64" and "arm64" being only two of many.5
\nMaybe you run heterogenous Kubernetes clusters like a mix of amd64 and arm64 based worker nodes. Or take Apple's new Macbooks into consideration, where the M1 family are ARM based, so for many years you'll have to deal with a potpourri of processor architectures in your company. The reasons are manifold.
\nIf you're lucky or not depends on why the image has only a single layer and how that also plays into the rest of your container environment. If you usually build most of your images from the same base layers and somebody decided to squash their image, then you might see yourself unlucky, because now you have to download the whole image and cannot skip the duplicated pieces anymore, and if such images are huge a lot of wasted time and space is what you get. But if you're dealing with let's say a bunch of single binaries wrapped in "FROM scratch" docker images, each of them very small, then there's no win in having shared layers, as there are none. Lucky you! 😉
\nLet's not get into the weeds of why it's commonly called amd64
even though it means "all 64-bit x86 processors" also known as x86_64
. The very short answer is that AMD was the first creating the 64 bit instruction set for the x86 architecture. But your Intel based processors will just work fine.
So the rule seems to be: if you're the first, you name the baby, or something along the line I guess.
To quote from a StackOverflow answer: »Never turn autocrlf on, it causes nothing but headaches and sorrows.«1 There's not much to add to that statement, so I provide some background to line breaks in general. No typewriters were harmed in the process.
\n\nIf you don't have to deal with CRLF delimited files, don't bother configuring git to automatically convert between the different line break options. Set autocrlf to false (and eol to lf) and live a happy life!
\ngit
is probably the most famous (and perhaps also most used) version control system (VCS) in software development. I had brief periods with CVS (not to be confused with CSV files) and Subversion (SVN), wanted to try Mercurial (but never did), heard of Perforce … but the tool I used most is still git. And it probably doesn't help that I put my projects on GitHub, and if I wouldn't there, it would be Gitlab or another git based hosting platform perhaps. Well, and at work we're basically in the same position.
And while git looks pretty simple at the surface it is anything but simple, or easy for that matter. A few months ago I watched a live stream by mgattozzi, where he walked through the git config documentation, took some time to cover all options. I learned a lot (and of course forgot most of it again).
\nSo if you ever think to yourself »I have been using this tool for decades and still can't get around it« then don't worry, we all have been there, and continue doing so.
\nMatters get a bit weird if you work across different operating systems which come with their own views on the world. And one of the constant troubles are default line breaks in text files. git
comes with some configuration options to accommodate the different needs.
Everytime you hit the ⏎ key2 in your editor and save the file, your computer writes down that line break.\nAnd unsurprisingly in tech we couldn't agree on, what's the best way do store such control character(s), so we had and still have different options.
\nFortunately the most common byte sequence nowadays on Linux and Unixoid systems (like macOS) is \\n
, also known as LF
for "line feed"; LF
is encoded as 0x0A
in hexadecimal, or 10
in decimal. It's also referred to as "newline" character.
Then there is Windows which loves its \\r\\n
sequence, called CRLF
, which is the combination of CR
and LF
. CR
stands for "carriage return" and has the value 0x0D
(13
in decimal).
A thing of the past is a single CR
, that's what Apple used in the pre-OSX/macOS times, but not anymore.
And to be honest, I wish we could convince Microsoft to also move to LF-only as a default line break control character. The savings on file sizes globally would be huge. But the very least uncountable to the preservation of people's sanity.
\nThat's a pretty historical reason and comes from the typewriter times. You know, those weird direct-to-paper keyboards, sometimes very mechanical and noisy. Well, we still have very noisy mechanical keyboards, but those can be connected to computers now. Why you would want to use one of those I cannot understand … but I digress.
\n\nTypewriters. You had a keyboard to enter your text, but the letters where always hitting the same spot in the center, so the paper moves from right to left to fill a line. And same also for multiple lines, you had to move the paper up.
\nFun thing is, you could do both operations, the vertical and horizontal movements, independently. Especially staying in the same line and moving back was great way to immediately correct mistakes and strike them out with a bunch of -
characters for example.
The carriage return was the movement back to the beginning of the (same) line; the carriage being the whole bulky thing the paper was placed in and moving horizontally while you were typing.
\nAs an aside: How did you know that you needed to return without looking at the paper (because you're supposed to look at whatever you needed to type up instead)? There was a bell integrated which rang when you hit some right margin, accustically telling you that you needed to finish your line.3
\nThe line feed on the other hand is the movement of the platen, that tube/roll thingy the paper was wrapped around. Usually it was moved enough to position the paper onto the next line you wanted to type.4
\nOf course, even in the age of typewriters, it was pretty common to do both actions in combination, therefore the carriage return lever was at some point in time usually a two-in-one part, used for both to move the carriage and trigger the line feed.
\nWhen typewriters became electrical devices they lost the lever which got replaced by the ⏎ key — it still carries the same visual meaning. But instead of moving some machinery physically down and left5, nowadays we only instruct our computers to position the cursor in the same fashion.
\nWithout digging this hole deeper I simply assume, that even printers in a text mode also needed and adhere to the line breaks your computer was sending to the device. To be honest I haven't looked into the lower levels of printing for a very long time. Most of the time I print some PDFs and I don't remember in which way such documents are sent over. Also it totally doesn't matter for this post. This is just my brain meandering into a few nooks and crannies, I'm sorry.
\n\n\n\nLoooooong story short: we inherited a system of ages ago, and are kinda-sorta still stuck with it. From typewriters over terminals to keyboards we continue to use the same old stuff over and over again. To make matters just a tad more complicated for us, Windows doesn't want to play with the others and assumes everyone is totally okay with different line breaks6, basically almost all programs and tools running on this OS will write files with CRLF if not told differently.
\nLuckily for us most programs do understand LF delimited files as well, editors and IDEs in particular are capable of dealing with them without any issues.
\nAnd if you program and develop across operating system borders, for example building software running on machines with some Linux, you most likely always want LF no matter where you do your work.
\nI made the mistake to follow advise on enabling git's feature to automatically translate between LF and CRLF, but it just doesn't really make sense if you ask me. If you're like me on a Windows 10 host, use WSL2, work with Visual Studio Code, and open projects either from the host or within a WSL2 environment, you are constantly faced with the different line breaks all the time. I also display the control characters with a plugin, so I actually see the difference. And it annoys me.
\nMy love for CRLF
was … never there in the first place.
Let's be honest, nobody is really confused about the idea that a line feed implies a carriage return. As mentioned in the footnotes, building stairs with text is probably the most uncommon thing, so removing this exceptional use case was quite okay from my point of view.
\nWhich brings us back to git
and it's options.
If you don't fall into some special bucket of »but I really must have CRLF encoded text files«, please never ever enable autocrlf
in your configuration.
Beyond other pretty useful defaults everyone should have the following lines in their global git configuration file:
\n# $HOME/.gitconfig\n[core]\n autocrlf = false\n eol = lf\n
\nUse autocrlf = input
if you still have to deal with files having CRLF in it and need to preserve it for whatever reason.\nAlso you might want to consider using safecrlf = warn
to be made aware of issues. But as far as I understand all of this is really only necessary if and when you have to deal with potentially mixed files (having both LF and CRLF). There might be totally valid use cases, but if possible just make a huge way around CRLF, avoid its path, forget that it exists.
You can also set the config on a per repository basis, but I would highly recommend to make it part of your global setup.
\n# If you don't want to touch the config file directly, run the following 2 commands;\n# remove `--global` if you're in a repo and want to change the setting only for it.\ngit config --global core.autocrlf false\ngit config --global core.eol lf\n
\nAlternatively you could also check in a .gitattributes
file into your repository:
* text=auto\n
\nAnd additionally to enforce a strict LF rule use .editorconfig
with the following configuration as a start:
# https://editorconfig.org/\n\n# top-most EditorConfig file\nroot = true\n\n# Unix-style newlines with a newline ending every file\n[*]\ncharset = utf-8\nend_of_line = lf\ninsert_final_newline = true\n
\nMost editors support editorconfigs either natively or via a plugin/extension. Highly recommended to establish shared editing standards.
\nAnd with all that stuff you should be fine. I hope. Windows always finds a way to surprise you.
\nI also invite you to read the git config documentation for further interesting options.
\nLast but not least you might be also interested in how to set up git for multiple profiles, which also can help you to maintain platform specific configurations, if you so desire.
\nSorry for this clickbaity title by the way. 😬7
\n\nFull answer: https://stackoverflow.com/a/2361321/653173
\nalso known and labeled as ↵ Enter, ↵ Return, or something similar; you find more keyboard shortcut versions at ⊞.
\nGuess what, we still have a control byte for the bell (BEL) in our ASCII/Unicode tables. It has the byte value 7 (0x07). Some terminal emulators (the software you use to interact with your shell) support that and can make a sound. Sometimes this is used to alert the user about errors.
\nSurely you could also just feed the line without a carriage return … if you wanted to make stair steps with your text. Not sure if that very useful. What I basically mean is, that there potentially very good reasons to have both actions separate. It probably covered the 100 % of all potential use cases.
\nYes, technically speaking everything was moving right and up, but your intention was still that you wanted to "move" your letter head even though that thing was not moving. There also have been other typewriter designs and the movements might be different there, but let's not go down yet another rabbit hole. ;-)
\nAnd Unicode transformation format, but that's totally out of scope here. Windows until version 10 still loves UTF-16 very much, while having some sort of UTF-8 support, but I hear that starting with Windows 11 they embrace UTF-8 a bit more, so in maybe 10 to 20 years we can at least agree on something in the wider (tech/operating system) world.
\nActually I'm not sorry at all. Was looking forward to the day I could use that in a blog post title.
\nGitHub can automatically delete merged head branches of pull requests only.1\nAnd I have a tiny GitHub Action for you to clean up the unmerged ones.
\n\nIn 2019 GitHub introduced a pretty nice feature, where you can set up your repositories to automatically delete the branch which just got merged after a pull request (PR) got accepted and closed. Which makes sense for most git workflows: your work just got into the main branch, so why would you want to keep the head branch of your feature around? Exactly. And yes, it is only a single click on a button in the interface, but you will be surprised how rarely people actually click on it.
\nSo, please enable the setting in your repo to free people from this burden, you'll find it on the main settings page:
\n\nOne problem we have at work is that in some of our repositories, there is a lot of work going on, people open PRs, most get merged, some don't, some branches and PRs propably get abandoned and forgotten … there are plently of reasons why such cruft builds up over time. And within a growing organisation this will just increase.
\nThis issue might get amplified by adopting a GitOps approach, where also more automation like bots create and manage changes and pull requests. And most likely not all of those PRs get merged.
\nSo while the successful PRs clean up after themselves, the unmerged branches still linger around.
\nAnd while branches are theoretically cheap (from a more technical point of view), they are not cheap for us humans. If I have a repository with usually 10 active engineers, but 30 or more open pull requests and an ever growing pile of branches of unmerged PRs and other work and experiments, this makes it harder for me to assess the health of the project. Are those 150 branches still needed? What is their state? Are they useful?
\nAlso for folks like me who get itching or even suffer from OCD, this becomes an unbearable situation.
\nFear no more, I have a tiny GitHub Action for your workflows. It is really just a few lines of code as it utilizes the github-script action, so we can focus on the important bits.
\nThe business logic is the following:
\nconst { repo, owner } = context.repo;\nconst pull_number = context.payload.pull_request.number;\nconst ref = `heads/${context.payload.pull_request.head.ref}`;\nconst checkParams = { owner, repo, pull_number };\nconst deleteParams = { owner, repo, ref };\n\nconst isMerged = await (async () => {\n try {\n await github.pulls.checkIfMerged(checkParams);\n return true;\n } catch (e) {\n if (e.status && e.status === 404) { return false; }\n // escalate all unexpected errors\n throw e;\n }\n})();\n\nif (!isMerged) {\n console.log(`Deleting branch: "${ref}"`);\n try {\n github.git.deleteRef(deleteParams);\n } catch(e) {\n console.log("Cannot delete branch; error:", e);\n }\n}\n
\nNot a lot, right?
\nAll it does is checking if the PR in question got merged, and if not then tries to delete the associated branch with it. I wrapped in a try-catch block, as you could have manually deleted the branch before the workflow ran, or if the git weather goddesses decided that it cannot happen for other reasons.2
\nYou can check and copy-paste the whole workflow file from the example repository over there:\nhttps://github.com/asaaki/gha-delete-unmerged-branches/blob/main/.github/workflows/delete-unmerged-branch.yaml
\nThe repo also showcases how it looks like when the action runs:\nhttps://github.com/asaaki/gha-delete-unmerged-branches/pull/2
\nAnd that's all there is to it.
\nOf course, you could also employ an Engineering Janitor, but nobody would want to pay for that. And to be honest: I have better things to do. 😜
\nThere are also more sophisticated GitHub Actions out there, but they do a lot more than what I need and want. Also for an organisation like at my workplace where also security is quite important we cannot use everything from the public interwebz without vetting it first. And instead of assessing a full blown external dependency the 20-something lines above will just do; at least it saves me and others a lot of time. So we can go back to building great products, while I am a happy guy again. 🐶
\n\nAnnouncement: https://github.blog/changelog/2019-07-31-automatically-delete-head-branches-of-pull-requests/
\nSince this is a clean up task and won't block you or others in any other way, there doesn't need to be done much more at this point. But feel free to file an issue if you believe the script needs to be more elaborate.
\nThis year I managed to write at least one—hopefully interesting—article per month.\nNow it's August, and I'm tired. Not only physically, but also mentally.
\n\nI even tried to start a newsletter as an alternative communication channel, but I haven't written anything for two months now.
\nIt's not the lack of topics, quite the opposite. I have an ever growing list of ideas and drafts. Some of them I wish to finish into something presentable. But I cannot find the energy and motivation right now.
\nAnd it's also not only because I think I'm not competent enough to talk about the topics.1
\nI have plenty of time, but at the end of the day I am so exhausted and drained, I only can do activities which do not require a lot of activity from my side. I binge-watch a lot, mostly series. I flee from the reality.
\nAnd I couldn't tell you what is causality and what correlation. What's the reason, and what's the effect.
\nIs it really just the pandemic? Or has it only revealed or amplified what was already there?
\nIs it only seasonal? Will it last? I don't know.
\nIt's not even like "typical" depressive episodes where everything is dark and gloomy. I'm really just low on everything. While everything does seem meaningless, I just sit there and think »I don't even care.«
\nLast year around this time I did a three months sabbatical. Today I would say, it was way too short. If I had planned it better (and oh boy am I bad at planning anything in my life), I should have taken a year off. So today would be my last day of a hypothetical one year long sabbatical.
\nWould it have changed something? I cannot tell.
\nI don't even know if I still love my job. Not that you need to "love" it, not hating it would be sufficient perhaps. But sometimes I wish I could time travel back 10 years ago, when all was new and shiny, and my biggest concern being if I can code that thing somehow.
\nToday I open my editor either only for a personal pet project (which I also seem to abandon now more quickly, and most of them don't even see the light of the world). Or to quickly check out someone else's pull request (PR) for local testing. I read a lot of code though. And review PRs. And faciliate, organise, manage. I am glue.2
\nLast year I had a great mentorship pairing, we enjoyed our exchanges. I learned as much as she did. I miss that.
\nI also miss not only coding, but also research and discovery. Time to throw around ideas and proof of concepts. I miss whiteboards.
\nMost of all I miss my energy, motivation, drive.
\nI could ramble a lot more here.
\nBut I'm just too tired.
\n\nIn fact it is, not gonna lie here.
\nAnd have no f*cking clue if I do my job right.
\nThis footnote is also tired.
\nLately I see people complaining about "the Rust community" to confuse memory safety with general safety and security. But do "they" really?
\n\nI'm neither a Rust expert nor a security expert. I am just yet another almost middle-aged white cis male mostly-straight-passing dude-like person in the internet.
\nBut maybe my amateurish view can still help to ease your frustration or reconsider your perception.
\nFirst: who are these Rust folks (the community) you talk about? Do you mean the language creators, the language maintainers, the different Rust teams, the crate authors and maintainers, the language and ecosystem users, the random person who just heard something about Rust, the game or the language?
\nI'm pointing this out, because if you complain about people being vague, fuzzy, and unclear in their communication, you cannot really make the same mistake. The Rust community is not a homogenous mass. It's a growing landscape of very diverse set of folks under the umbrella of Rust, the programming language.
\nAnd I am mentioning that, because—as far as my own memory serves me ;-)—I have always tried to emphasize that Rust is not a magic silver bullet for everything, that the language only tries to protect you from a specific subset of problems, but a very prominent and important one nevertheless.
\nCompanies like Microsoft and Google ran some analyses and realized, of all security vulnerabilities they got reported, around 70 % can be attributed to memory safety issues.1
\nThis is indeed an important take away and why it should be in the toolbelt of your arguments if you want to nudge people into the Rust direction. In other words, it would be a waste if you don't mention that.
\nJust to repeat: I still only talk about a subset of problems where Rust can support you.
\nEven if you do not want to rewrite all the things to Rust (which is totally fine by the way!), as long as you adopt at least the same or similar mindset, take notes, and maybe even copy some approaches to make your language and ecosystem more memory safe, we all win. As far as I can tell the C++ folks do that and try to implement stuff in the language to make it safer. I really love that these two communities even talk and collaborate with each other, as some people believe they have to be sworn enemies.2
\nSo my first assumption: On the way so far people simplify, generalize, and cut out important pieces, so that others now see that weird message like »Rust makes everything safe and secure.« Which is of course a lie.
\nRust cannot prevent the other very important problem space: logical errors. The compiler and the tools are not a magical being which can look into your mind and guess what you intended to write and tell you the right way. Nope, it will happily accept your hundred lines of bullshit as long as it makes sense to the type system and its rules.
\nYou can write code which compiles and does nothing. Or anything but the thing you want it to. As long as you don't mess up the memory, you're free to mess up everything else. Go, have fun!
\nTo be clear: You are responsible for the business logic of your program!
\nAnd you know what helps to write code which does what you expect it to do, now and in the future? Writing tests! Yes, I said it, you still have to write tests, even in Rust.
\nAnd the beauty of it is, that you really can focus on testing your (business) logic of your code, because as we learned, for the memory safety issues the compiler already has your back.3
\nFor the security aspect I cannot say much, but I consider that to be part of a bigger test umbrella anyway. I've heard of fuzzing for example, basically robustness testing, where you automatically bombard your program with a lot of random inputs to see if you missed something. The bugs revealed might be of any category, and yes, they can still be memory safety issues, even in Rust land.
\nWhy? Because the Rust compiler and tools are also just programs, code written by people. And we all make mistakes or miss things. And so the compiler can produce faulty code. Most of the time though it does not compile at all if that happens, still very annoying, but probably safer than compiling and crashing.
\nSo given that even the compiler might have flaws, nobody can give a 100 % guarantee that Rust will never have or produce any memory safety issues. At least I wouldn't want to bet on that with my life.
\nI do not know anyone who actually has this misconception and does miscommunicate in that way, that suddenly "memory safety == (100 %) safety" or "memory safety == (full) security" is a thing. I have to trust the voices brought into my Twitter timeline that they met somebody who did.
\nAs far as I can tell nobody I know and is close to the language would do that mistake. Definitely not intentionally. But also not really by accident.
\nAlso communication is a two-ended thing, it is possible that you misinterpreted something those folks said to you. I cannot tell, because nobody told me their full story so far. So my assessment is solely based on those ranty messages and my own interpretation of what might have gone wrong.
\nBut rest assured, we "the Rust community" are not all the same, some of us do understand the subtle differences between memory safety, safety in general, and security.4
\nAnd since this article was sparked by a specific complain about a "memory safe replacement" (please
) for a SUID program (sudo
): Of course, software written specifically for security purposes should definitely have more extra care, no matter if written in Rust or any other language.5
I do not believe that Rust's advertisement of being a more memory safe language is inherently harmful for security. But then again, I try to reason with a common sense, yet also get reminded, that we have warning labels like »contains hot drink« on the lid of a to-go paper cup full of hot coffee. So maybe we need a big warning sign on the Rust language homepage? »You still need to think for yourself!«
\nDepending from where you're coming from Rust does have a great type system which also can help to prevent quite some logical errors as well. But you have to write that stuff, or find somebody who writes it for you. But the idea here is, that if you create good types for your business domain, the compiler can also enforce some rules you would have to write tests for otherwise. No matter what you do, you still have to think and write something for it. Even if you have a Copilot. 😉
\n\nMicrosoft: https://msrc-blog.microsoft.com/2019/07/16/a-proactive-approach-to-more-secure-code/; Google Chrome, Chromium project report: https://www.chromium.org/Home/chromium-security/memory-safety; I find the Chromium report interesting, because half of the memory issues are use-after-free, something safe Rust prevents you from doing, and most of your conversations with the borrow checker will be about that. Embrace them!
This is a simplification, not generalization, there will always be some humanoids hating each other simply for their language choice, and that is just sad. To be fair, I also don't like all languages and ecosystems, but I try to channel that purely into the technical thing, not the people. Sometimes I might fail, but I pride myself to be usually pretty good at separating language and their users.6
\nYet that does not mean, that I'm also always okay with how some language communities act and think and operate.7
\nSome folks close to me try to remind me, that I should assume good intent in all actions. Sounds nice on paper, but I believe this is also a fallacy, sometimes a dangerous one. But I digress already too much in my footnotes. Follow me back to the main thread, would you?
\nAs long as you do not remove all the guard rails with too much unsafe
here and there. And when you do (have/need to) use unsafe
you might want to consider writing just some more tests. Bam!
Or do have some vague idea about the differences at least.
\nAfter I followed some trails I also don't really understand the outrage anyway, I cannot see that the please
author actually claimed that their Rust rewrite is 100 % secure, only that it is a memory safe replacement. There was a security code review, issues were identified, and were or will be addressed in the future. So from a distance it looks all like quite fine open software development to me. Of course, I do not know if and what kind of conversations might have happend behind the scenes. (And therefore I purposefully have not linked to anything in this whole blog post nor named names. I don't want to make it more personal than it needs to be.)
I don't know who needs to hear that, but meritocracy is not a desirable goal. It is not fair, it creates inequality, and it is harmful to inclusion and diversity. It is a dystopia and satire.
\n\nIf you believe in meritocracy I do not want to work with you!
\nI simply do not have time for your elitist and exclusionist world view.
\nIf you're interested in more thoughts, take this door.
\nCoincidentally others also seem to be thinking about it right now:
\n\nAlso: Happy Pride Month! 🏳🌈
\n"},{"id":"https://markentier.tech/posts/2021/06/whiteboards/","url":"https://markentier.tech/posts/2021/06/whiteboards/","date_published":"2021-06-05T00:00:00Z","banner_image":"https://markentier.tech/posts/2021/06/whiteboards/cover.png","summary":"Something missing in my life is that huge blank canvas and a marker in my hand. And a team around it to draft ideas and be creative. A tiny love letter to whiteboards. 💌","tags":["whiteboard","canvas","pen","marker","collaboration","team","physical","virtual","digital","tool","creative"],"content_html":"Something missing in my life is that huge blank canvas and a marker in my hand. And a team around it to draft ideas and be creative. A tiny love letter to whiteboards. 💌
\n\nRecently I stumbled upon this quite long list of thoughts of someone on Reddit:
\nDrunk Post: Things I've learned as a Sr Engineer
\nI find it quite interesting, many items resonate with me. Of course, there are also things I disagree with, and when you read it you will probably, too.
\nBut I think there are some items worth spending more than just a second or two.
\nAnd one of them was something I felt and feel too hard right now: the lack of whiteboards and the collaborative effects which come with them. Yet as most reasonable companies should do, I work from home and see my coworkers only virtually.
\nNow people mentioned it in the comments of that reddit post, and we also do in our team and organization: there are tools like Miro, which are basically digital whiteboards and canvases to brainstorm, create, think, draft, work together. But to be fair I feel the same as the author, it is simply not the same for me.
\nMiro is a great tool and it helps me to digitalize my brain dump, but it cannot 100 % replace the look and feel I want, I need for the first step of my creative process. I still use pen and paper to scribble and write things down before they enter a computer, I have plenty of sticky notes around my monitor. I even have tiny, portable whiteboard plates, though I don't use them as often as I'd like to.
\nAnd being in the same room with my colleagues, scribbling, connecting, erasing, and throw around ideas is something no digital tool has satisfiably replicated for me.
\nSo yes, I miss office life, and one of the reasons is this kind of collaboration. And no, I am not asking anyone to even try to reproduce this digitally and virtually, I simply want to go to a physical place, stand in front of this physical board, hold a real marker in my in hand, smell the chemicals from it, embrace the haptics and the feels … and most importantly enjoy this moment with my team mates.
\nI'm looking forward to a world, where all of this is possible again.
\nI hope, nobody asks me how I feel about whiteboard interviews …
\n"},{"id":"https://markentier.tech/posts/2021/05/self-care/","url":"https://markentier.tech/posts/2021/05/self-care/","date_published":"2021-05-31T00:00:00Z","banner_image":"https://markentier.tech/posts/2021/05/self-care/cover.png","tags":["care","self-care","guilt","health","mental","mental health","toxic","work","workplace","company","culture","job"],"content_html":"I don't think anyone should feel bad about leaving bad and toxic workplaces.\nWhile it is lovely if you care about the people you leave behind,\ndo not forget: you always have to take care of yourself first.\nOtherwise you won't be able to help others in the future.
\nSadly enough I have to remind myself about this. Although I probably try to care about others and other things to distract myself about the fact, that I should follow my own advise every once in a while, too.
\nDon't get me wrong, my current workplace is not toxic and I don't think about leaving the company. The above statement of the first paragraph was a response to a talk at this year's Euruko (a Ruby conference), the speaker mentioned that she felt guilty about leaving a place and thus leaving great people behind.
\n(And nowadays people can also suffer without an inherently harmful workplace. This pandemic has revealed a lot about ourselves, society, and culture.)
\nLater in the Discord chat the direction of the conversation shifted more towards if and how people can leave, but this is a completely new and huge topic to have.\nJust very briefly: I admit, I am extremely privileged1 and can leave any time I want and also find very easily a new place to work, now with more remote-friendly companies out there even more easily than before. I know that not everyone is in this luxurious position. The only thing I can give to the less privileged: when you're completely burnt out, you will also suffer and probably loose your job, so don't wait too long for a change and try to escape that situation which is destroying you rather sooner than later. And talk to people, don't suffer silently, don't be too proud to not even admit to your very own family you try to care for and to protect, that you hurt. They probably need to know first anyway.
\nWhatever you do, take care! Self-care!
\n\nI am single, I am white, I am male, I am German, working for a German company, I am well paid, I have zero responsibilities and can spend my time and money for anything I want. All the things I am not or where I am different do not matter here.
\nAfter reading some documentation of some PowerToys utilities I noticed that Microsoft found a pretty neat trick to use a unicode character as symbol for their Windows shortcut key: ⊞ Win
\n\nMany people might be familiar with the command key ⌘
found on Apple keyboards, which makes it easy to denote keyboard shortcuts for Macos applications. In Unicode this is actually the "Place of Interest Sign" (U+2318), which is the origin for the symbol at Apple, in case you wondered if Apple had bought itself into the Unicode standard.1
So one had to wonder if and when Microsoft would do that for Windows applications. I cannot really tell you anything about the history of the unicode character usage, but I find it remarkable enough that they have found a (quite hacky) way around it by utilizing the unicode character Squared Plus in documentation and help. Some might say it's kind of abuse of it, as it says "squared plus" and not "window" or something similar, it's in the mathematical operators block, so some people might find it even offensive?
\nBut even Wikipedia itself states:
\n\n\n Wikipedia uses the Unicode character U+229E ⊞ SQUARED PLUS as a simulation of the logo.\n \n\n
I say hacky, because it's math symbol, but on the other hand, Apple also didn't do really a much better job at it for their key.
\nAlso Microsoft couldn't have used the current ⊞ Win for very long, since their past logos looked different and had no lookalike character in the Unicode standard at all. Only with Windows 10 the logo became so stylized and abstract enough that it could be simulated with the ⊞ character. So it couldn't have been earlier than 2015, but I guess the actual appearance made it even much later.
\nSince the usage of such symbols is not very common, you also cannot easily type it with your keyboard, unless you have remapped it somehow — people at Microsoft surely have, right?
\nIf you need it quickly right now, here for you to copy it: ⊞
\nSome more meta data about this symbol:
\nName | Squared Plus |
Category | Math symbols |
Since | Unicode Version 1.1 (1993) |
Block | Mathematical Operators (U+2200 - U+22FF) |
Plane | Basic Multilingual Plane (U+0000 - U+FFFF) |
HTML entities | ⊞ ⊞ ⊞ |
CSS | \\229E |
UTF-8 | 0xE2 0x8A 0x9E |
UTF-16 | 0x229E |
Other nice keyboard shortcut symbolization:
\n↹
)Note: the F1 – F12 keys can also be used without (media) symbols, but since some keyboards emphasize the media controls more than the function key names, it might be more usable to add the symbols in your documentation.
\nIt's interesting to see that there are some dedicated ISO keyboard symbols, but I cannot recall to have seen all of them in real life so far. Sometimes a few of them are supplementary to the text labels.
Consult standard ISO/IEC 9995-7
for more details, if you have it at hand.
Funnily enough I'd initially planned to post a very short post about the ⊞ Win only, and then it turned into this slightly more elaborate piece. I could even dig deeper into some of the other examples here. If you're interested in something in particular or know more common or uncommon keyboard symbols, let me know and send me a message on Twitter.
\nRead a short story at https://en.wikipedia.org/wiki/Command_key#Origin_of_the_symbol
\nSee https://en.wikipedia.org/wiki/Any_key#Cultural_significance; I refer to the space bar key simply because it allows to put a custom text label on it.
\nBut feel free to use it anyway.
\nNot long ago I replaced my homepage of an image based overview with a very slim Table Of Contents version.\nThe trickiest part was to make that responsive. I finally found a solution. As usual: no JavaScript involved.
\n\nA table of contents (TOC) page is something coming from the good old book world. Here's an example:
\n\nIt lists the different chapters and on which page you can find them.\nFor this kind of presentation there is a slight usability problem if it hadn't been designed the way it is: the chapter titles are pretty short for the given font size and therefore a lot of space between the title and the page number would occur.\nNow guess how hard it would be to find the correct page for each chapter without accidentally jumping a line or two. I've had my fair share of such listings with smaller fonts and lots of items. More than once I took a ruler or pen to use it as a guide. Not a very pleasant experience.
\nTo help the reader not getting lost publishers usually add those pretty useful dotted lines which connect the chapter title and the page number visually. Now there is no ambiguity about what belongs to what.\nThese dotted lines are more commonly called »leaders« or »leader lines« as they lead your eye from one end to the other. Besides dotted leaders there could be also dashes or underscores or any other symbol, but the dot seems to be the leading sign for this feature.
\nAs with a lot of letterpress practices and habits this kind of contents page presentation also got adopted into the web world. Such index pages are not too common, but I have seen them over the years. At the time of writing my current homepage of this blog is also sporting a TOC overview for my articles; instead of page numbers I use the publishing date on the right side.
\nI have iterated over that page quite a lot recently, mainly because I have a responsive design and want that contents listing adjust appropriately.
\nNow the actual tricky part are very long titles which then wrap over multiple lines. This can happen either at all times or only at certain breakpoints, because font size and width would make the titles wrap.
\nContents\n==================================================\n\nA very short title ............................. 1\nSlightly longer title which needs to break into\nmultiple lines to fit on the page .............. 2\nAnother short title again ...................... 3\n
\nWhile this might not impose a huge problem in word processors and the print world, the web is still a bit behind when it comes to fancy presentation styles.
\nIt took many years to get flex boxes1, grids2, and multi-column3 layouts.4
\nMaybe one day we get a specialized table of contents layout into CSS as well …
\n<!-- not working - just an idea how the future could look like -->\n<style>\n .toc { display: toc; toc-fill: dots; }\n .chapter { toc: chapter; }\n .page { toc: page; }\n</style>\n<div class="toc">\n <div class="entry">\n <span class="chapter">The ultimate answer</span>\n <span class="page">42</span>\n </div>\n</div>\n
\n[Update 2021-03-16]: There is a CSS3 draft spec including leaders5, but since March 2011 (so even 10 years later) no further development happened to get actual support into browsers. The draft proposes content: leader(dotted)
, which would avoid the long and hardcoded string, but a lot of the following solution would be more or less identical, unless further proposals would also land. (source)
But since we do not have that (yet … ever?), let's build our own contents page with a flexible layout.
\nSome more or less usable results can be found in this StackOverflow question, but I believe the following solution is the most modern and condensed version. I'm stil very grateful for all the prior work done by the responders there and elsewhere in the internet.
\nOur example HTML we want to style could look like the following:
\n<div class="toc">\n <h5>Contents</h5>\n <div class="entry">\n <div class="chapter">A chapter title</div>\n <div class="page">1</div>\n </div>\n <div class="entry">\n <div class="chapter">Another awesome chapter</div>\n <div class="page">5</div>\n </div>\n <div class="entry">\n <div class="chapter">Be more creative with your titles ;-)</div>\n <div class="page">9</div>\n </div>\n</div>\n
\nFor brevity and focus I use only <div>
s here. On my actual homepage I used <h2>
headings and <time>
elements and lots of other microdata, which is not important for this example.
Right now that wouldn't look very interesting:
\nEverything is just one after the other in a single column.
\nNow let's make that more to something like a tabular view. But instead of a table, let's use the more flexible CSS Grid system:
\n/* each entry line will become its own tiny grid */\n.entry { display: grid; }\n
\nAnd since we already know that it has a 2 column layout (chapter on the left, page number to the right), we can also prepare that:
\n.entry {\n display: grid;\n grid-template-columns: auto max-content;\n grid-template-areas: "chapter page";\n}\n
\nThe first column should automatically take the remaining space (auto
) which is not occupied by the page number (max-content
).\nWe're also giving the areas some names for easier reference. We only have a two by one grid, therefore a single line for the areas description is needed.
We also need to tell our pieces where they should go:
\n.chapter { grid-area: chapter; }\n.page { grid-area: page; }\n
\nIf you don't use grid area names, the code would look a bit different here:
\n.chapter { grid-area: 1; }\n.page { grid-area: 2 / 3; }\n
\nI find it a bit more confusing, because the numbers are not the grid cells, but the column (or row) edges between them, starting with 1
at the left (or top) outer edge.
┌───────────┬────────┐\n │ │ │\nstarts here -> 1 chapter 2 page 3\n │ │ │\n └───────────┴────────┘\n
\nYou can read about all the details of this property on MDN.
\nAt this point we already have some usable intermediate result:
\nBut we're still missing some pieces. The leaders are not there yet, and the page numbers also need to be aligned with the last line of the chapter title (if that is a multi-line text).
\nThe alignment can be fixed with the align-items
property. Also let's be good citizen and add a gap
between the chapter and the page divs, so the dot leaders are not crashing into the numbers in the worst case. While you could also use paddings and margins, the gap
property of the grid also does a very good job at it.
.entry {\n /* ... */\n align-items: end;\n gap: 0 .25rem;\n}\n
\nFor the leaders we can use the ::after
pseudo element. This has the advantage to not only add them without extra HTML markup, but it avoids unnecessary content, which could and would be parsed and processed by machines otherwise. Not that dots are harmful, however also not really useful either.
.final-chapter::after {\n content: " . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . "\n ". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . "\n ". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ";\n}\n
\nYet this alone would render horribly. We have way too many dots than actual space to be filled.\nWhy on earth so many, you might ask? Well, at least you would need to find the minimum amount of dots (and spaces) to occupy the shortest possible chapter title in your smallest font size with the biggest contents table width. Long story short, if you need more, duplicate the middle line once or twice. The above amount did serve me well so far, I would hope it would also be sufficient for you as a starter.
\nIn an earlier version I used some background properties with a radial gradients for the dot rendering. And before that even some embedded SVG dots. First I thought that this new way seems hacky and verbose, but in the end I used less code and markup for the job. And I'm pretty sure that a continuous string of spaces and fullstops compresses well on the wire.
\nHow to trim the dotted line to the needed length?
\nTo be honest this also took me a while and lots of googling and »stackoverflowing.«
\nThe answer is to use overflow
as well as some good portion of position
usage.
.final-chapter {\n /* ... */\n position: relative;\n overflow: hidden;\n}\n.final-chapter::after {\n /* ... */\n position: absolute;\n}\n
\noverflow: hidden;
cuts all content overflowing the whole chapter div, this does not affect the chapter title itself, as this particular part is text which can wrap over multiple lines, but it will apply to the ::after
content thoughposition: relative
on the parent (here chapter div) is only to support the next one:position: absolute
is a bit difficult to explain; its possible use is broader than what we need here, but the important piece is that it does not reserve extra space for the element, so the overflow rule kicks inThe rule position: absolute
usually sounds scary and in most cases works the way that it will then position the element absolute to the page. But this is actually not the full truth.\nIf you have an ancestor with a position of relative or absolute, then this closest container will become the reference for absolute positioning. This is a neat trick to move around a child in a parent div with the usual top/bottom and left/right properties instead of awkward padding and margin shuffling.
For the left-to-right text scenario our positioning rules can be interpreted as »oh, look there is some room until the end of the container (right side of the last line of text) , so write the ::after
content there, but cut off everything which does not fit« … exactly what we need here.
| <-- max available width --> | .page ↴\n ┌─[ .chapter ]────────────────────────────────────────────┐ ┌─────┐\n │ ┌─[ text ]───────────────────────────────────────────── │ │ │\n │ │ "The first line of my very very verrrrry long chapter │ │ │\n │ └────────────────────────────────────────────────────── │ │ │\n │ ──────────────────┐ ┌─[ .chapter::after ]───────────────┐ │ │\n │ title overflows" │ │ free space for the dot leaders │ │ 123 │\n │ ──────────────────┘ │ (overflow limits to .chapter box) │ │ │\n └─────────────────────┊───────────────────────────────────┊ └─────┘\n ┊ »invisible dots land« ┊\n └┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┘\n
\nNow we have all the pieces for a responsive leaders layout.
\nWithout further ado everything together in the …
\nWith some tweaks as follows:
\nA live example can be seen here, and the code is also in this repo.
\n<div class="toc">\n <h5>Contents</h5>\n <div class="entry">\n <div class="chapter">A chapter title</div>\n <div class="page">1</div>\n </div>\n <div class="entry">\n <div class="chapter">Another awesome chapter</div>\n <div class="page">5</div>\n </div>\n <div class="entry">\n <div class="chapter">Be more creative with your titles ;-)</div>\n <div class="page">9</div>\n </div>\n</div>\n
\n.final-entry {\n display: grid;\n grid-template-columns: auto max-content;\n grid-template-areas: "chapter page";\n align-items: end;\n gap: 0 .25rem;\n}\n.final-chapter {\n grid-area: chapter;\n position: relative;\n overflow: hidden;\n}\n.final-chapter::after {\n position: absolute;\n padding-left: .25ch;\n content: " . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . "\n ". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . "\n ". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ";\n text-align: right;\n}\n.final-page {\n grid-area: page;\n}\n
\nDon't want to type the long content string yourself and you use sass for your project?
\n@function repeat($char, $times) {\n $content: "";\n\n @for $i from 1 through $times {\n $ccontent: $ccontent + $char;\n }\n @return $ccontent;\n}\n\n.chapter::after {\n /* ... */\n content: repeat(" .", 80);\n}\n
\nOn a related note, the Smashing Magazine has a nice collection of some Table of Contents of the letterpress world. Maybe you find some inspiration for alternative index page designs there if the classical TOC gets too boring.
\nI hope this walkthrough was useful to you.
\nHave you tried it yourself? Is something not working as expected? Let me know and send me a message on Twitter.
\nHappy table of contenting! 📖
\n\n\nIn fact I am so old that I do remember the time when even CSS was not a thing. <table>
s where our main tool for layouts, <font>
and <center>
our design spice … let's <marquee>
roll some text again, shall we? </marquee>
I work across 2 computers and 3 OS, all with nearly same development setup. For work I need some special care for my git config, since I want/need access to personal repositories.
\n\nRecently I helped a coworker during onboarding with his machine setup and he asked me about this very same topic. I provided him with what I have and use. The following blog post is a refined version of it.
\nThe slightly more complex version control system configuration is due to the following requirements and expectations:
\nI know, I know, this looks like a huge list of requirements. But not only is this my personal preference, partially also my workplace has some strictler rules, especially when it comes to security.
\nIf you do work in an early stage startup you might think that it would be excessive, but I promise you: if that company becomes successfull and grows and turns into a bigger deal like an enterprise, things will change. And part of it will be around security and compliance. You better improve your setup now when it's still easy to do so.
\nOn Latacora you can find a nice compilation of things you need to consider and implement if you want to go big and sell your startup, but a lot of it could and should be done without that goal in mind.
\nSo, if you cannot get rid of the first point and do need also access to your personal persona on GitHub (or any other git based hosting platform for that matter), this article might help you to achieve it.
\n»Why would I continue using my personal GitHub account on my work machine anyway?« — You may or may not be an open source contributor, and you may or may not use such open source software at your current employer's projects for example. The reasons are plenty and some will justify the need for what is described here.
\nBy the way, this approach should work for both single and multiple GitHub accounts. Since GitHub makes it easy to keep using a single account while also being a member of an Enterprise organization, I haven't bothered testing it with a true multi-account configuration, but since you will use individual SSH and GPG keys for either way your computer won't really know the difference.
\nWhile git
does support the HTTPS transport, in most cases you will use the more preferred way of talking git+ssh instead. So let's tackle this lower level first:
If you working with GitHub you want to generate keys with the latest and greatest recommended algorithms:
\nssh-keygen -t ed25519 -C "your_email@example.com"\n
\nIf you work on other platforms, please check first, which algorithms are supported there.
\nIn decreasing order of preference and security:
\n# prefer this if possible:\nssh-keygen -t ed25519\n\n# this is still quite okay:\nssh-keygen -t ecdsa -b 521\n\n# this only as a last resort option, should work everywhere:\nssh-keygen -t rsa -b 4096\n\n# avoid this, please;\n# also GitHub does not allow new keys with DSA anymore!\nssh-keygen -t dsa\n
\nMore details about key generations at: https://www.ssh.com/ssh/keygen/
\n~/.ssh/config
### github\n\n### -- PERSONAL/MAIN ACCOUNT --\n\nHost github.com\n Hostname github.com\n User <YOUR GITHUB USERNAME>\n IdentityFile ~/.ssh/<YOUR PERSONAL SSH KEY>\n\n### -- WORK PERSONA/ACCOUNT --\n\nHost github.com-work\n Hostname github.com\n User <YOUR GITHUB USERNAME>\n IdentityFile ~/.ssh/<YOUR WORK SSH KEY>\n\n### general\n\nHost *\n AddKeysToAgent yes\n IdentitiesOnly yes\n PreferredAuthentications publickey\n Compression yes\n
\nImportant note: Do not change the order of the configuration.
\n\n\n\nFor each parameter, the first obtained value will be used. The configuration files contain sections separated by ''Host'' specifications, and that section is only applied for hosts that match one of the patterns given in the specification. The matched host name is the one given on the command line.\n\n
Since the first obtained value for each parameter is used, more host-specific declarations should be given near the beginning of the file, and general defaults at the end.\n \n
Since we will use two different hosts, we must repeat the Hostname
line, but also want to specify User
and IdentityFile
specifically. If you use your ssh config only for git and also have only a single user name, you could move that config line into the generic section in the bottom. Though I still prefer the explicit repetition for each specific Host block.
At this point you might wonder »How is github.com-work supposed to function?«\nHold that thought, we will come back to it later.
\nYou want to use GPG for signing commits.\nYour workplace might not require it (yet), but if you have any level of trust issues, or just want to be sure that a commit was made by you and be able to prove it, use commit signing.
\nInterestingly GitHub does not recommend you to use a more modern algorithm and requires you to pick RSA. A bit sad, but RSA with 4096 bits seems to be still fine for this purpose.
\nAfter you have created your keys, you should grab the IDs for them:
\ngpg --list-secret-keys --keyid-format LONG\n\n# I use shorter IDs and git doesn't seem to struggle\ngpg --list-secret-keys --keyid-format SHORT\n
\nExample output:
\n/home/asaaki/.gnupg/pubring.kbx\n-------------------------------\nsec rsa4096/D73D7242 2021-02-14 [SC]\n AE93078BDC15BF6A84767DBBA3CBBB61D73D7242\nuid [ultimate] TEST KEY <test-key@example.com>\nssb rsa4096/57047776 2021-02-14 [E]\n
\nThe ID is the part after the rsa4096/
from the sec
line: D73D7242
In the LONG variant it just used more characters from the whole fingerprint (which you can completely see in the line between sec and uid).
\nThe git
command line interface (CLI) improved a lot over the years. I remember that I used a custom GI_SSH=…
wrapper in my shell environments to make this whole host/key mapping possible, but thanks to the power of git configs this is a thing of the past.
~/.gitconfig
[init]\n defaultBranch = main\n\n[commit]\n gpgsign = true\n\n# other sections cut for brevity; add the following to the bottom:\n\n[user]\n name = Your Name\n useConfigOnly = true\n\n# optional (if you sometimes work outside of your regular directories)\n[include]\n path = ~/.gitconfig.personal\n\n[includeIf "gitdir:~/Development/Personal/"]\n path = ~/.gitconfig.personal\n\n[includeIf "gitdir:~/Development/Work/"]\n path = ~/.gitconfig.contentful\n
\n~/.gitconfig.personal
[user]\n email = your.personal@email.here\n signingKey = <PERSONAL GPG SIGNING KEY ID>\n
\n~/.gitconfig.work
[user]\n email = your.work@email.here\n signingKey = <WORK GPG SIGNING KEY ID>\n\n[url "git@github.com-work"]\n insteadOf = git@github.com\n
\ninclude
and includeIf
are the key components to separate out specific configurations based on where you are on your system.
include
is always pulled in, this is useful if you really want to separate out parts of your main .gitconfig
.
If you only work in specific directories like ~/Development/Personal/
and ~/Development/Work/
— and this means where you also need to commit and push — then you could remove the general [include]
section entirely. I keep it around, because I might have checked out a repository somewhere else and don't want to move it for a commit.
There's probably a better way to organize this, but the above has served me well for quite some time now.
\nI recently added the global [user]
section for setting useConfigOnly = true
and the name, since I only have one. useConfigOnly
prevents git from guessing name and email and forces it to read it from the configuration files. If the email would be missing, git will complain the next time you try to commit or push. And you will know that something is broken in the configuration.
Important note: The trailing slashes (/
) on the [includeIf …]
lines are very important. If you forget them, then git would try to match only this very specific folder and ignore it for any folder within it. More details about conditional includes in the git documentation. (I totally missed that you can use them even for branches, too.)
The signingKey
values should be set appropriately based on the IDs you have noted from the previous section of this article. Now when you commit anything git will use the correct key based on where the repository directory lives.
To automatically enforce the commit signing, use commit.gpgsign
set to true
.
Remember the question you had earlier about the weird looking Host configuration in SSH? The [url "git@github.com-work"]
section is the counterpart making it work, because git will do the translation when you are in your company's repos.
What will happen is the following:
\ngit@github.com
in it with the value of the [url …]
, here git@github.com-work
.github.com-work
host and tries to find all matching configurations for that, including the exact one we have defined above.github.com
does not match anymore.Hostname
instead of Host
(translating it back again).What's with this [init]
block you haven't mentioned before?
Since Git 2.28 you can set a name for the default branch when creating a new repository. Be a good citizen and avoid offensive and negatively connotated terms. The wider developer community seems to like "main" as a good replacement for the previous default name. You can read more about renaming existing default brances on GitHub.
\nYou might want to use passwords for both the GPG and SSH keys, your employer might even have a rule for this. To avoid annoyances in your workflows make sure your system has some sort of keychain manager and can keep the passwords for a period of time.
\nSince the setups vary for each operating system and this part is also out of scope for this post, I leave it to you to figure it out.
\nIn total 2 files to change and 2 files to add (excluding all the keys), and you have a nice setup for using multiple accounts, emails, GPG keys, and/or SSH keys on your computer.
\nAnd from here you can expand it even further if you like or need.
\nLast but not least: this now works for existing repositories, locally created, or freshly cloned ones. As long as the repos are somewhere nested in the right parent directory, you're set and done.
\nTo quickly check which config is used execute the following command in a repository:
\ngit remote -v\n
\nThe URL should provide you a quick hint if the correct profile is used.
\nAlternatively you can also run:
\ngit config --show-origin --get user.email\n
\nThis will also print where the final value was retrieved from.
\nGit came a long way and I'm very glad and happy that we can have such setup without a lot of manual tinkering and workarounds. Gone are all my custom scripts, snippets, and .envrc
's for that purpose, which were also not completely platform agnostic.
Have you tried it yourself? Is something not working as expected? Let me know and send me a message on Twitter.
\n"},{"id":"https://markentier.tech/posts/2021/01/rust-wasm-on-aws-lambda-edge/","url":"https://markentier.tech/posts/2021/01/rust-wasm-on-aws-lambda-edge/","date_published":"2021-01-18T00:00:00Z","banner_image":"https://markentier.tech/posts/2021/01/rust-wasm-on-aws-lambda-edge/cover.png","summary":"Ever felt limited by the languages on AWS Lambda@Edge? Wanted to run Rust for your CloudFront triggers, but re:Invent 2020 disappointed you in that matter? Let me show you one way of how you can still get it done.","tags":["rust","wasm","webassembly","aws","lambda","lambda@edge","cloudfront","serverless","amazon","trigger","edge computing","CDN","Node.js","wavelength","amazon web services","wasm-bindgen","wasm-pack"],"content_html":"Ever felt limited by the languages on AWS Lambda@Edge? Wanted to run Rust for your CloudFront triggers, but re:Invent 2020 disappointed you in that matter? Let me show you one way of how you can still get it done.
\n\ntl;dr:
Check out the repo and give it a spin for yourself. You can also jump ahead to the code part and skip the story section.
At the last AWS re:Invent 2020 the company with a smile did announce quite some changes, improvement, and new products/services. Notably also a lot of interesting stuff in the AWS Lambda space. Yet one area—one very dear to me—was left out completely, total radio silence, nothing, nada: no improvements or new features for Lambda@Edge (L@E). This made me very sad. 😢
\nI tinkered with a custom solution in the past and also got some experiments running, but it was a pretty hand-rolled approach and nothing I could give to others if they wanted to do the very same.
\nI sat down yet again and took some of my learnings into a repository, so we have a starting point for some Wasm based serverless apps on the edge.
\nAs of today (January 2021) AWS only offers two languages for Lambda@Edge: Python
and Node.js
. Both are available in relatively decent versions, too.
Since I have never bothered with Python too much in my life, but have my fair share of experience with JavaScript (JS) and node, I stuck to the latter as the trampoline environment for the setup. Also I know that it comes with WebAssembly support out of the box, which wouldn't be the case for Python anyway.
\nAs you've maybe guessed the project will be some Rust, which gets compiled down to WebAssembly. This artefact then can be loaded and executed in the node environment.
\nA word about performance: I have not evaluated a "JS vs Wasm" comparison, and it's also not part of this exercise. There have been people and article in the past vouching for one side or another, all with their own benchmarks. So I won't bother you with that and advise you to take your own measurements.
\nWebAssembly will not beat all JavaScript code, especially very fine-tuned one. V8 (the underlying JavaScript engine for both Chrome and Node.js) is a very performant beast and comes with just in time (optimizing) compilation for further boosts.
\nThe Rust code in Wasm clothing can give you probably certain garantuees you miss from JavaScript, but again you have to evaluate if the benefits are worth the effort.
\nPotentially you might also consider to switch to Python as your runtime instead. At least that language should have real integers as far as I know. 😉
\nNo doubt you can build and deliver very fast and also safe programs with Rust/WebAssembly. Especially if you need some specific algorithms and computations where JS/node might not be the best and you would resort to some (probably C-based) native libraries anyway.
\nThere are only a few issues with that:
\nYou have not full control of the execution environment of your edge functions. Sure, you can introspect with a test function what you're dealing with, but how sure will you really be, that the environment on CloudFront does provide the exact same system and system dependencies as your local development environment (or the non-edge Lambda environment for that matter)? AWS has a track record of not providing you with reproducible local environments. In fact, it looks like they get away with it even further, since the announcement for containerization support for regular AWS Lambda. People, who know me, also know that I'm not a big fan of big docker images, but I'm afraid that's what we will see now happening there. I hope AWS promotes good containerization guidelines to prevent that waste-of-storage mess. Furthermore I really don't want to see docker on the edge for that reason. One can just hope, right?
\nYou work in a very constrained environment. Check the current limits for Lambda@Edge: the zipped code can use up to 50MB on the origin side and only 1MB max if it shall be deployed for the viewer side. Of course, this is usually still plenty of spaces for most use cases, packaging up plain JS results in very small archives. But once you take the first issue into consideration, then this could actually become another problem for you.
\nThe size restriction can be mitigated for JS-only code pretty easily by bundling the code with WebPack, Parcel, or Rollup. General advise is anyway to never deploy the unoptimized package especially when you want to push it to the edge. The node_modules
folder grows very big, can still have quite some bloat even after an npm prune --production
, because it only looks at packages, but not the content of them. Yep, my minimalism brain kicked in again.
The system dependency problem can only be solved by using solely pure JavaScript libraries and packages. That might work for a while, but eventually some use case might demand a non-JS solution (either a native library or some executable).
\nFor example let's say you want to build an image transformation function and want to use sharp
, a very well-known package in the JS ecosystem, then you already end up with around 37 MiB of data in your node_modules folder alone. Zipped up it's still around 13 MiB. That might be enough for you to run it as a trigger on the origin side of your CloudFront distribution; it's just about showing you how quickly a node project can grow.
Whatever your reasons are, I hear you.
\nAWS is improving on one side, but also losing it on another.\nWhen it comes to CDNs (Content Delivery Networks) and Edge Computing, the competition is now sprinting ahead of AWS.
\nI cannot say a lot about Fastly's offering, it's mostly behind some beta doors, and mere mortals like myself are not allowed to peek. They have their fastlylabs, but that's for experimentation, not the final offering. So I don't even bother to check it out.
\nI can tell a bit more about Cloudflare though, because their edge computing offering is available and affordable to small businesses and individuals (even free in most cases). Check out Workers, really do! I have already played around with Workers and KV storage, it's a quite pleasant experience. I might write about a setup for them in the future as well.
\nGitHub repository to follow along:
\nhttps://github.com/asaaki/rust-wasm-on-lambda-edge
\n$ tree -L 1 -a -I .git # output truncated for brevity\n\n.\n├── .github - GitHub Actions workflow and dependabot\n├── .gitignore\n├── Makefile - very convenient make targets\n├── README.md\n├── fixtures - simple test fixtures and script\n├── node - Node.js wrapper\n├── rust - Big Business Here!\n└── <and some more …>\n
\nzip
to package up the function for uploadOn your machine you need to install Rust, node, wasm-pack, and zip, if not present yet. The workflow for GitHub Actions has that already sorted out for you.
\nThis article won't give you steps to get your local development environment set up, please use a search engine of your choice and look up how to do it.
\nI adopted a rollup based approach, since it's quite easy to get configured and also something we use at work. I always found webpack a little bit too cumbersome, and parcel is just yet another new kid on the block. I'm pretty sure you can adjust the project to your needs. All we need here is: compile TS to JS and bundle up everything into a single JS file. In the past I found the WebAssembly dependency management very tricky, in the end I used a plain "move the .wasm file into the final bundle" approach, which just works fine, because I did not want to inline the WebAssembly code (as most plugins try).\nMaybe you have a smarter solution for that, please open a pull-request in the repo. Just keep in mind: wasm-bindgen creates already a pretty decent module loader, so there is no need to work around that, but I fail to get any of these bundlers to move the wasm files along with it into the bundle output directory.
\nI use TypeScript here, because it gives you some nice support during development. Also the aws-lambda type definitions were useful to create the Rust structs and adjust the serialization. (AWS is actually very strict about the JSON you pass around, "something": null
for absent data does not work, either it is a completely required field with a strict type like a string, or it should not be in the JSON at all).
In general for any bigger frontend or node backend project I would recommend to use TS nowadays. While not every of your dependencies might come with type definitions, at least within your own codebase you can enforce strict rules and type checks.
\nTo make the node wrapper as slim as possible, we pass the event and context data directly into the WebAssembly and return whatever it returns.
\nBtw if you do return a specific struct instead of a generic JsValue
, then TS checks will also kick in and use the auto-generated type definitions from the wasm-bindgen process.
For a quick baseline test and project I did not go down that road yet, as it would require to replicate all the TS specific stuff in the Rust code (wasm-bindgen cannot do everything fully automated yet). This is a great opportunity to create a utility crate for that, basically like rs2ts but in reverse direction. I wish aws-lambda-events already had those CloudFront events in it, but sadly they don't.
\nYet also be aware of certain type limitations, read on to learn more about them.
\nThe Rust code is also nothing special so far.
\n// src/lib.rs\n\n#[global_allocator]\nstatic ALLOC: wee_alloc::WeeAlloc = wee_alloc::WeeAlloc::INIT;\n\nmod types;\n\nuse std::panic;\nuse types::cloudfront as cf;\nuse wasm_bindgen::{intern, prelude::*};\nuse web_sys::console;\n\ntype JsValueResult = Result<JsValue, JsValue>;\n\n#[wasm_bindgen(start, final)]\npub fn start() {\n panic::set_hook(Box::new(console_error_panic_hook::hook));\n console::log_1(&intern("(wasm module start)").into());\n}\n\n#[wasm_bindgen(final)]\npub async fn handler(event: JsValue, _context: JsValue) -> JsValueResult {\n console::log_1(&intern("(wasm handler request call)").into());\n let request = cf::Event::request_from_event(event)?;\n\n // TODO: Fancy biz logic here ...\n\n request.to_js()\n}\n
\nNote: The displayed code might not be up-to-date with the repository version.
\nThere is one function (start
) which is triggered when the Wasm module gets loaded. You can use it to set up some internal state if needed. We only used it here for configuring the panic handler; whenever an unrecoverable error happens, it gets logged via console.error
, helps immensely with debugging. And as we do console logging anyway, there shouldn't be any significant overhead for that part. The compilation output will probably a bit bigger because it needs to store more information for the better panic output.
The other—probably way more interesting—function is handler
, which takes the inputs from the JS side, does … a lot of nothing, and returns a request JSON blob for CloudFront to deal with.
Currently the machinery reads the arbitrary JsValue
and tries to deserialize it into a struct, so we can deal with it in code. This is definitely not the most efficient way of doing it, but the conversions in and out really avoid some current existing pain points.
For example wasm-bindgen has not a great story around Rust enums, for now only very simple C-style enums are allowed. Meaning: for our CloudFront (CF) event data, which can be more strictly typed into either a CF request or response event, this does not play well with Rust's richer enums, as we cannot convince wasm-bindgen to use them. There is an open issue around this topic, but it was created just recently and thus no work has been done yet. Similarly Rust's Vec
is also not fully supported yet (see issue 111), which might be the even bigger issue for some of us.
Workarounds can be a lot of Options and serialization skips, as I do internally anyway.
\nSome transformation overhead can be addressed by using serde-wasm-bindgen, but in my example repo I'll use it only for the input side (deserialization). On serialization a collection like HashMap or BTreeMap gets turned into an ES2015 Map, which is unfortunate as well, because they cannot be JSON stringified.
\nAs you can see, currently there are trade-offs to be made in all corners, but that shouldn't stop us to explore further.
\nIn the current state of the project I have provided pretty strict structs and enum for the CloudFront event data, it even surpasses now the TypeScript interfaces, which makes my point from the previous section pretty obsolete now. I still wish it was easier to autogenerate Rust types from TS definitions. The only good thing about CloudFront related data is, that it won't change that much … if at all. Some APIs in AWS have been stable for years now, so a "write once, use forever" approach might be sufficient.
\nI tested against a simple CloudFront distribution with S3 origin, within that bucket a small static HTML file to be served.
\nI live in Berlin, Germany, my closest AWS region is eu-central-1
(Frankfurt), and the CloudFront POP is usually FRA2-C1
as well.
I have pretty stable and fast connection, 100Mbit/s or more in download speed with a ping around 20 to 30ms are common for me.
\nKeep in mind: the following numbers are only true for me. You might observe completely different performance. Therefore I also recommend to measure the different baselines.
\nAll tests will use the following command:
\nwrk -d 30 -c 5 -t 5 -R 5 -L https://<MY_DISTRIBUTION_DOMAIN>/\n
\n-d 30
- run it for 30 seconds-c 5
- only 5 connections, it's not supposed to be a heavy load test-t 5
- just aligning it to the connection count-R 5
- rate (throughput) of 5 (requests/s), also to avoid unnecessary contention-L
- latency statistics; I will only use the shorter summaryThe numbers are fairly low because I don't want to test the overall performance of CloudFront in my area, but want to get consistent and repeatable numbers for L@E in general. Under high load the performance is impacted by many other factors, which we cannot really control.
\nTo ignore the cache, all functions will be deployed as Viewer Request triggers, "Include body" enabled to give it a full round picture (no body payload will be sent and used though).
\nFurthermore after deployment I will run a warm-up round to eliminate the cold-start period, I'm not interested in those bad numbers. I know they are horrible, but we cannot do anything about it really; eventually a function will tear down and a new one needs to be spawned. So each wrk
will be run twice, but only the second run will be used here.
Bare CloudFront distribution with S3 origin, small HTML file as index (~1.4 kB filesize).
\nRunning 30s test @ https://<MY_DISTRIBUTION_DOMAIN>/\n 5 threads and 5 connections\n Thread calibration: mean lat.: 29.903ms, rate sampling interval: 56ms\n Thread calibration: mean lat.: 29.821ms, rate sampling interval: 53ms\n Thread calibration: mean lat.: 29.696ms, rate sampling interval: 51ms\n Thread calibration: mean lat.: 30.232ms, rate sampling interval: 65ms\n Thread calibration: mean lat.: 30.708ms, rate sampling interval: 66ms\n Thread Stats Avg Stdev Max +/- Stdev\n Latency 23.39ms 1.30ms 29.10ms 84.00%\n Req/Sec 0.98 4.00 20.00 94.23%\n Latency Distribution (HdrHistogram - Recorded Latency)\n 50.000% 23.09ms\n 75.000% 23.73ms\n 90.000% 24.64ms\n 99.000% 29.04ms\n 99.900% 29.12ms\n 99.990% 29.12ms\n 99.999% 29.12ms\n100.000% 29.12ms\n
\nLet's remember a p99 of around 30ms.\nDepending on internet weather it varies a bit, ±5ms are not uncommon.
\nEmpty here means only that the handler is not doing anything.
\n'use strict';\n\nexports.handler = async (event) => {\n return event.Records[0].cf.request;\n};\n
\nYep, that's all: take the request object out of the event and pass it down. It's the smallest possible function you could write. It would also be the most useless, too.
\nRunning 30s test @ https://<MY_DISTRIBUTION_DOMAIN>/\n 5 threads and 5 connections\n Thread calibration: mean lat.: 43.793ms, rate sampling interval: 81ms\n Thread calibration: mean lat.: 47.265ms, rate sampling interval: 106ms\n Thread calibration: mean lat.: 44.360ms, rate sampling interval: 90ms\n Thread calibration: mean lat.: 44.558ms, rate sampling interval: 84ms\n Thread calibration: mean lat.: 44.164ms, rate sampling interval: 81ms\n Thread Stats Avg Stdev Max +/- Stdev\n Latency 37.21ms 3.76ms 56.29ms 87.00%\n Req/Sec 0.97 3.15 12.00 91.21%\n Latency Distribution (HdrHistogram - Recorded Latency)\n 50.000% 36.32ms\n 75.000% 37.95ms\n 90.000% 40.19ms\n 99.000% 49.98ms\n 99.900% 56.32ms\n 99.990% 56.32ms\n 99.999% 56.32ms\n100.000% 56.32ms\n
\nSo the baseline with Lambda@Edge being active for the p99 is now 50ms. On rainy days basically twice as slow as without any triggers. Again, account for some variance around ±5ms.
\nThe reported duration in the Cloudwatch logs for the function is between 0.84ms and 1.53ms, so on average around 1ms simplified. This begs the question, where the other overhead went. There are roughly 20ms unaccounted for and missing. A tribute to the performance gods? I don't know. 🤷
\nJust keep that gap in mind. I guess this is the overhead between the Cloudfront request handling and call out to the L@E execution environment, somehow all this stuff needs to be orchestrated behind the scenes. It's just sad that I cannot find those timings anywhere. The pure CloudFront logs are also not conclusive.
\nThe maximum memory used is 67 to 68 MB.
\nSee the code in the repo, I used the exact same version of it.
\nRunning 30s test @ https://<MY_DISTRIBUTION_DOMAIN>/\n 5 threads and 5 connections\n Thread calibration: mean lat.: 44.980ms, rate sampling interval: 90ms\n Thread calibration: mean lat.: 50.256ms, rate sampling interval: 167ms\n Thread calibration: mean lat.: 44.296ms, rate sampling interval: 92ms\n Thread calibration: mean lat.: 45.668ms, rate sampling interval: 87ms\n Thread calibration: mean lat.: 49.321ms, rate sampling interval: 154ms\n Thread Stats Avg Stdev Max +/- Stdev\n Latency 36.71ms 2.23ms 47.78ms 82.00%\n Req/Sec 0.95 2.82 11.00 89.07%\n Latency Distribution (HdrHistogram - Recorded Latency)\n 50.000% 36.38ms\n 75.000% 37.34ms\n 90.000% 38.62ms\n 99.000% 44.32ms\n 99.900% 47.81ms\n 99.990% 47.81ms\n 99.999% 47.81ms\n100.000% 47.81ms\n
\nNo, the Wasm function didn't get magically faster than the "no-op" node version. With a p99 around 45ms it is just in the same ±5ms corridor.
\nWe can conclude from this, that the module has no significant performance hit.
\nReported duration is usually somewhere between 1.25ms and 1.50ms. I haven't seen it dropping further below, so let's say there is a 250µs on top of the average node baseline.
\nThe maximum memory used is between 75 to 77 MB. I guess this is the additional allocation for the WebAssembly module, yet I'm not too worried about that. I assume the overhead can be amortized by running more memory efficient code within the module instead of the node environment. I'm pretty sure that a plain old JavaScript object needs more memory than a Rust struct.
\nThis is all great news: you can run WebAssembly on AWS Lambda@Edge without a noticeable performance penalty. Now write your Rust code and run it on the edge.
\nOf course I do hope that in the future this will become more native. There's a lot of development happening in the WebAssembly space.
\nBut maybe I've also convinced AWS to not move any faster, because we can solve the problem ourselves. And for a behemoth organization like them it can take many years to deliver even the smallest improvements which we consider to be no-brainers, … and then we wonder why it took them so long in the first place.
\nYet I stay optimistic in general. I know that they know that Edge Computing is some hot stuff right now. They even launched a very specialized offering called AWS Wavelength. I'm looking forward to test this once it's more widely available.
"},{"id":"https://markentier.tech/posts/2021/01/rust-most-popular-language/","url":"https://markentier.tech/posts/2021/01/rust-most-popular-language/","date_published":"2021-01-09T00:00:00Z","banner_image":"https://markentier.tech/posts/2021/01/rust-most-popular-language/cover.png","summary":"Someone regularly reminds me of this fact. So that I do not forget, I write it down.","tags":["rust","language","programming","popular","most loved","stack overflow","survey","R.E.S.F.","evangelism","strike force"],"content_html":"Someone regularly reminds me of this fact. So that I do not forget, I write it down.
\n\n\n\nDid you know that Rust is consistently voted the most popular language on Stack Overflow for 5 years?\n \n
Yes, Flo, I do know, thanks to you. 😉
\n2021-01
Flo is a prophet, he predicts that it will also be true for the sixth year in a row. Yet I will hold back with updating this post until the 2021 survey was conducted and evaluated.
The reason for this running gag is that there are also a lot of blog posts and articles out there just stating the same fact without really providing much more value. The Rust community and surrounding observers never get tired to parrot the same phrase into pure meaninglessness. The only way to take it is with humor, of course, similar to what we experienced with the »Rust Evangelist Strike Force« (R.E.S.F.), a group which wants to port everything to Rust. For some folks I'm probably considered to be part of it. 🤷🏻♂
"},{"id":"https://markentier.tech/posts/2020/11/microsoft-eating-software-world/","url":"https://markentier.tech/posts/2020/11/microsoft-eating-software-world/","date_published":"2020-11-09T00:00:00Z","banner_image":"https://markentier.tech/posts/2020/11/microsoft-eating-software-world/cover.png","summary":"Usually the answer to such indicative title questions is No, but perhaps this time the answer is … Maybe yes?","tags":["Microsoft","Windows","WSL","WSL2","software","TypeScript","JavaScript","GitHub","VSCode","Visual Studio","Visual Studio Code","code","Codespaces","Windows 10","Windows Terminal","PowerShell","PowerToys","Azure","Azure Pipelines","Azure DevOps"],"content_html":"Usually the answer to such indicative title questions is No, but perhaps this time the answer is … Maybe yes?
\n\nI host my code on GitHub.\nI edit my code with VSCode.\nI program in TypeScript sometimes.\nI use the Windows operating system.\n And the Windows Terminal.\n And the WSL.\nI check sites on Edge every once in a while.*\n\nI have not used Azure yet, though.\n And the many other open source projects\n or all of their paid software.\n\n\n*) That is more a point for Google nowadays, isn't it? 🤷🏻♂️\n
Let's have a high-level view on their catalogue of some products. Not an exhaustive list, but showing the most pertinent to the topic of software development.
\nTo be fair here, GitHub (GH) was only an acquisition, but from Microsoft's (MS) point of view probably a very good one indeed. If you like it or not, but GH is the code hosting platform, based on the equally famous version control system called git.
\nAnd since GitHub also has now Actions (CI workflows), MS didn't even need to buy another competitor separately. Sure, GitHub Actions is not always on par with other CI/CD platforms, but for most people it will do the job.
\nMore about Azure DevOps/Pipelines further down.
\n🔗 https://code.visualstudio.com/
\nWhile "Visual Studio" is the big brother, a full featured IDE, mainly used for Windows application development (though can be used for other use cases like web as well), Visual Studio Code (VSCode or simply Code for short) is only "little" on the surface. It is slimmer and thus very fast. The main focus is probably on web development, since it ships with some extensions targeting this field by default. Like the following …
\nTo be honest, after having used and tried so many editors (Notepad++, vim, Textmate, Sublime Editor, Atom Editor), this one finally stuck. The vast amount of extensions makes it a power tool for your every day coding business.
\n🔗 https://github.com/features/codespaces
\nWhen GitHub and VSCode had a child, then it would be it.
\nCodespaces is basically a full development environment in the cloud. You edit your GitHub hosted project in a browser version of Visual Studio Code and can run compilation or other build steps in a containerized system (using Docker).
\nCurrently still in beta, but I hope it will get widely accessible soon. I got my access pretty quickly (and overall the first time I got into a beta at GitHub anyway). While Codespaces is not necessarily for me (I have a beefy home PC and love to run a lot locally), people working on less powerful machines and/or on the way/remotely will enjoy such environment. Due to its tight integration of components the experience is much smoother than you might know from alternative cloud based IDEs.
\nMaybe if you work on a Raspberry Pi 400, then Codespaces is probably the perfect environment for you.
\n🔗 https://www.typescriptlang.org/
\nIf JavaScript (JS) was a strongly typed language it would probably have been TypeScript (TS). There were other attempts like Facebook's Flow in the past, but in the end TS won. (At least I haven't heard anyone talking about anything else but TS.)
\nOf course, it is not natively supported by browsers or NodeJs, but it is not too difficult to compile your .ts
files to JS. And with Deno you have the option to run it without this step in between. (And don't worry, while they want to remove the internal TS code, that does not mean, deno will not support TS for your projects; read this nice summary)
Long story short, if you want to more confidently develop JS based applications and services for either frontend or backend, you should have a look at TypeScript.
\n🔗 https://www.microsoft.com/windows
\nI have to admit, Windows 10 is probably the most usable version of the Windows operating system since the time I had my first PC with Windows 95 (yeah, I got lucky and skipped the 3.x times). There are still the regular "Oh, here's an update and you have to restart" events, but at least compared to MacOS it is not even that much more anymore.
\nWindows alone wouldn't do the cut though, so let's also look more into the ecosystem around it.
\nI think this one of the most surprising developments driven by Microsoft itself. Sure, we have virtualizations (VirtualBox, VMWare), we have Docker, but actually a much tighter Linux integration into Windows? I never imagined that coming. Yet they did. And I'm very grateful for that. Especially with WSL 2 most of the things got a bit better and faster, only the filesystem performance across the boundaries needs to be fixed, as you can read in my previous article about my struggles with git status under WSL 2 and how to fix it.
\nEven Docker for Windows itself can piggyback on WSL 2 if present and configured. Which also allows for interesting cross OS workflows, since you can access your containers from Windows and Linux at the same time.
\nAlso the networking is done well, so that you can spawn a webserver in your WSL session and open in your browser of choice on your Windows host. Such smooth experience is not self-evident everywhere.
\nIt's like iTerm2 on MacOS or one of the gazillion terminal emulators under Linux. It is not extremely feature rich, but it is fast and slim, looks great, and handles my PowerShell & WSL sessions easily. You should consider that over the standard terminal which looks as ugly as the cmd.exe CLI.
\nAnd while Alacritty (GH; a cross platform terminal emulator) is also an option, it lacks tabs. Since I have multiple sessions open (at least one WSL and one PowerShell) I want and need them grouped into one application window.
\nDo you remember the good old cmd.exe
console? Yeah, PowerShell is nothing like that. It's a pretty decent shell. Since I'm still more used to Linux/POSIX like shells I stick to Git Bash for quick "on-Windows" work or open a WSL2 environment instead. On the other hand with pretty nice Rust based tools I can have them on both Windows and Linux environments and thus also call them quickly from a PowerShell, too.
I would have dreamed of such an environment back then in the late 90s/early 2000s!
\n\n\n\nPowerShell is a cross-platform task automation and configuration management framework, consisting of a command-line shell and scripting language. Unlike most shells, which accept and return text, PowerShell is built on top of the .NET Common Language Runtime (CLR), and accepts and returns .NET objects. This fundamental change brings entirely new tools and methods for automation.\n \n
That's a mouthful description. Since I haven't done anything with .NET, I cannot tell if that is a good thing to have.
\nThat is more a convenience toolbelt, but I mention it, because it provides some tiny niceties to the Windows experience, like better application window management (my actual main use case for installing it). Given how obsessed some Linux users are with their tiling window managers, PowerToys might have some good competition in its bag to make a switch at least possible.
\nIf you're a Windows power user and not using PowerTools yet, go check it out now!
\n🔗 https://azure.microsoft.com/
\nI know basically nothing about this cloud computing platform, but it is somehow not surprising that Microsoft runs one, since they also have moved a lot of their software offerings to the cloud in the first place. While many people and companies may work on AWS and GCE, some might want to consider Azure simply to have a one-stop shop for all their MS based development work.
\n🔗 https://azure.microsoft.com/services/devops/pipelines/
\nProbably more interesting would be to look at Azure Pipelines, which is interestingly in direct competition to GitHub Actions. I wonder if and when both CI platforms merge into a single solution, but given that it's part of the whole Azure DevOps I doubt that would happen anytime soon.
\n🔗 https://www.microsoft.com/edge
\nSince they changed their underlying engine to Blink and V8 (so yet another Chromium based browser like Chrome, Opera, and other derivatives), Edge is not a very special browser at all. I don't use it often, only if I need a Chrome like experience without incognito mode, but still without my Google account attached to it. I still have a Firefox installed as well, for the true cross browser checks.
\nIt's interesting how Microsoft actually finally gave up in the "browser wars", defeated by the overwhelming lead of Google. Older folks might remember that MS were the leader once with Internet Explorer, but lost it due to an extremely flawed browser everyone hated.
\nI don't know about you, but seeing the impressive list of tools, services, and products Microsoft has released over the last few years, a lot of them even open source, makes me wonder if I could do my professional job fully on a Windows based system, too. (For personal stuff I already do.)
\nCurrently I work on a Macbook with MacOS at my job, and the only difference is the more native look and feel workflow for Linux/Unix based development. On the other hand, we use containerization via Docker a lot, setting both Mac and Win on the same level anyway. For plain work all relevant tools and application exist for all 3 major operating systems.
\nI'll definitely continue using my Windows based home PC, having the Linux installation purely as an emergency fallback, but not relying on it for my daily tasks.
\nOf course, I will never convince hardcore Linux users to switch, ever. But I might have given you a brief overview what has happened in recent years in the Microsoft and Windows ecosystem, and not everything is so dire as it was like 10 years ago. I remember the times I tried to do web development with the LAMP stack (the XAMPP project seems to be still alive and kicking), it was a nightmare sometimes, and also one of the reasons I switched to Linux for quite some time, having experienced different distributions and their package managers; I even tried a Linux from scratch setup once.
\nAlso there is still the entry question in the room …
\nSo will Microsoft eat the software world? Or have they done already?
\nIt's a yes and no.
\nYes, they have strong stands in some key areas of software development, you can rely on a lot of Microsoft provided apps and services. I haven't even discovered and used all of them.
\nNot even Apple comes close to that, where almost nothing is for free or open source anyway.\n(And they don't build a lot in-house, their open source page is a joke.)
\nBut also No, because there are also areas where they are not the leading force. When it comes to Web topics I consider Google a much stronger competitor and more influencing than MS. That earlier mentioned browser turf war is probably a hint (which is topic for another day).
\nYet being in control of how the web apps are build (TypeScript) and which tools are used for that (VSCode, GitHub) is probably still a good counter move so far.
\nFurthermore there is the follow-up question: Would it be really bad if Microsoft ruled the software development world?
\nAnd to that I have not a good answer yet. Right now I welcome the change, as it is also very beneficial to me.
\nIt is also on the same level with Google's dominance in the Web and Mobile development. So both topics might be worth a look on their own.
\nFun fact: I haven't used any office suite for years, and it looks like Microsoft went to the cloud without me noticing it.
\nWhat's your opinion about Microsoft's push in the software development field? Do you like it, hate it? Let me know and send me a message on Twitter.
\n"},{"id":"https://markentier.tech/posts/2020/11/raspberry-pi-400/","url":"https://markentier.tech/posts/2020/11/raspberry-pi-400/","date_published":"2020-11-08T00:00:00Z","banner_image":"https://markentier.tech/posts/2020/11/raspberry-pi-400/cover.png","summary":"Raspberry Pi 400 was released, an (almost) all-in-one desktop computer kit. And the new package just reminded me of my first computer, the Amiga 500.","tags":["raspberry","pi","Raspberry Pi 400","kit","rpi","rpi400","Amiga","Amiga 500","desktop","computer","keyboard"],"content_html":"Raspberry Pi 400 was released, an (almost) all-in-one desktop computer kit. And the new package just reminded me of my first computer, the Amiga 500.
\n\nOn announcement day and afterwards my Twitter timeline was pretty dominated by the announcement of the new device, the Raspberry Pi 400. And first I felt even a bit annoyed by it, which might be more about the pandemic and isolation speaking than an actual dislike about the news. Because thinking about it a bit more I start to like the idea behind the new computer quite a lot.
\nMy first "real" computer (besides an kids edu device when I was eight years old) was an Amiga 500, which is quite similar to the Pi 400. And I have fond memories about that time. Of course, back then and now, both have their advantages and disadvantages.
\nOn the plus side: the computer is in the keyboard; this helps to save space at home, no place for a bulky PC case needs to be found. Similar to a laptop this is generally quite convenient. If you have a TV with HDMI at home, you don't even need to buy an extra monitor, the mouse is in the kit, so you should be able to get started pretty quickly. With the Amiga it was not that easy, but we got it with a suitable monitor anyway. Also it was much bigger than the Raspberry Pi 400, but still fit on a desk. Keep in mind, in the 90s we had to deal with huge CRT monitors which usually took quite some space on the table. Everything was still manageable.
\nOn the other hand: the computer is in the keyboard. Wait what, wasn't it just an advantage a minute ago? Yes, but it can be a negative point, too. I haven't had neither a Raspberry keyboard, nor do I have access to the Pi 400 kit yet. Depending on who you ask, the keyboard is quite decent. But if you have preferences (like want to have a clickety-clackety mechanical one) you might dislike the Pi 400
alltogether, but would maybe love the Amiga 500
instead. Also if you love a numpad, the Pi keyboard might not be your thing, too.
A more complete and nice review about the device can be seen on Jeff Gerling's YouTube channel, or if you like written form, then his blog post might be more appealing to you. But also ExplainingComputers had a look at it … as did many, many other people out there. So I won't list them all.
\nBy the way, as mentioned in Jeff's teardown video, the name »Raspberry Pi 400« seems to be a homage to the Amiga 500 (and other similar computer in a keyboard devices). So my memory trigger was not by accident then. Though in the design article they seem to have loved the Commodore 64 slightly more; I never had one sadly.
\n\nThe new Pi might not be for everyone in general, besides the computer in a keyboard aspect mentioned above there are probably also more reasons why to choose it or have a pass on it.
\nI clearly see it for two areas/groups:
\nBeginners can be kids or other people not having a chance to play and use computers before. And of course, while talking about kids the class room would be also a nice fit for the Raspberry Pi 400.
\nDue to its price point around $70 this hardware it an very affordable piece for schools or low/no income households. A complete kit (Pi, mouse, power supply, HDMI cable/adapter, book, SD card with OS) will set you off $100 (or roughly 97€ / £94, depending on seller), plus some monitor (which can be cheaply bought on eBay or second hand, let's say $30 to $50 for a decent one). So for under $150 you can get a full work station. If you can spend a bit more, add at least an USB hard drive to store your data in a better place than the SD card.
\nAnd also school boards should have an easier time to equip there classes with computers now with that pricing.
\nFor tinkerers and hobbyists like me the new Pi might not be a go-to device in general. Though I could imagine to have at least one on my desk attached via KVM switch to my screen, just as a more stationary pedant to my other berries floating around in the flat: as a test bed to try things before they get deployed to the others, the compilation station, … see, there are some use cases. Yet I will probably not purchase one right now, but just have it in the back of my mind once I get bored again. Which rarely happens.
\nAnd I'm more of a blue berry guy anyway. 🤷🏻♂️
\nIf you want to buy one, head over to the product page and select your desired keyboard layout and country, then they will provide you with some distributors.
\nAlso: Will the Raspberry Pi 400 make the »Year of the Linux Desktop« finally happen?
\nAnd why don't we have a raspberry emoji yet?
\n"},{"id":"https://markentier.tech/posts/2020/10/medium-icon-svg/","url":"https://markentier.tech/posts/2020/10/medium-icon-svg/","date_published":"2020-10-15T00:00:00Z","banner_image":"https://markentier.tech/posts/2020/10/medium-icon-svg/cover.png","summary":"Medium got a new logo. And vector graphics are kinda my thing. So I took a look at their icon's SVG source. It's pretty small, but you know, it can be shrunken further.","tags":["svg","scalable vector graphics","vector","graphics","symbol","icon","logo","brand","medium","2020","unfinished","ellipses","logomark"],"content_html":"Medium got a new logo. And vector graphics are kinda my thing. So I took a look at their icon's SVG source. It's pretty small, but you know, it can be shrunken further.
\n\nWhile I don't particularly like their redesign—I haven't really recognized that icon as an (typographic) ellipsis—the bigger issue for me was that for such a simple symbol they used a too complicated approach. I don't know their internal design process, but I suspect there was not an SVG aficionado involved. Especially not one with a strong urge to opimize the last bit out of such assets.
\nMy first impression in a tweet:
\n\n\n\nNot that I really care much about them anyway, but my first reaction was \"is something broken?\" I checked even the source code to see a simple SVG icon. (And why they went for a full path instead of 3 circles …? 🤷🏻♂️)\n \n
So let's take a look at the beauty.
\nThe code is not horribly big, too. So I have to give them that.
\n<svg viewBox="0 0 1043.63 592.71" class="aa bb">\n <g data-name="Layer 2"><g data-name="Layer 1">\n <path d="M588.67 296.36c0 163.67-131.78 296.35-294.33 296.35S0 460 0\n 296.36 131.78 0 294.34 0s294.33 132.69 294.33 296.36M911.56 296.36c0\n 154.06-65.89 279-147.17 279s-147.17-124.94-147.17-279 65.88-279\n 147.16-279 147.17 124.9 147.17 279M1043.63 296.36c0 138-23.17 249.94-51.76\n 249.94s-51.75-111.91-51.75-249.94 23.17-249.94 51.75-249.94 51.76 111.9\n 51.76 249.94"></path>\n </g></g>\n</svg>\n
\nWith linebreaks and whitespaces removed the file still weighs in at 479 bytes. (I did not count the xmlns, since they embed it directly in HTML page without it. Otherwise the file size would go up to 514 bytes.)
\nIt's still a mouthful of path data if you ask me. Also a lot of decimal numbers, even for the viewBox. Why?
\nI mean, what do you really see when you look at it?
\nThere are 3 (geometrical) ellipses there, right? Can we agree on it? Good.
\nGuess what, within the Scalable Vector Graphics (SVG) specs there are more ways to draw things on your screen than just complex paths.
\nFor example there happens to exist elements like <circle>
and <ellipse>
. What a surprise! Such primitives are quite easy to use, and probably also easier to understand when you look at SVG source code.
For comparison my very quick & dirty optimized version of the Medium icon:
\nAnd the corresponding source for it:
\n<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1770 1000">\n <circle cx="500" cy="500" r="500"/>\n <ellipse ry="475" rx="250" cy="501" cx="1296"/>\n <ellipse cx="1682" cy="502" rx="88" ry="424"/>\n</svg>\n
\nWith linebreaks and whitespaces removed the file only consumes 199 bytes. That's only ~42 % of the original size. (I added the xmlns only here; unless you embed the SVG in your HTML, most browsers do not like it without it. Remove 35 more characters for a 1:1 comparison, ergo 164 bytes, ~34 %.)
\nSome things to note here:
\nI removed unnecessary elements like the <g>
groups, which do not carry any useful properties. I'm pretty sure the class="aa bb"
stuff does not really need to be there.
I tried to match the radii as close as possible. Given that the source used a path and I had to eye-ball it, there could be tiny teeny bit of mismatch when overlaying both versions. It's a very quick replica, done in a few minutes. Right now I'm not sure if the source has perfect circle and ellipses.
\nI scaled the image up to a box with a round big integer number (using the height as reference and set to 1000). The reason is to avoid decimal numbers. Even if you try to round to only a single fractional digit, you still waste 2 bytes (including the point). And if you do that for basically all your values, you have quite some overhead which might not really add any value.
\nLearning: If you have primitives, do not convert them to paths!
\nIsolated example from my circle shape above:
\n<!-- simple shape -->\n<circle cx="500" cy="500" r="500"/>\n\n<!-- converted to path -->\n<path d="M1000 500a500 500 0 01-500 500A500 500 0 010 500 500 500 0 01500 0a500\n 500 0 01500 500z"/>\n
\nYes, sometimes a path could lead to less, like in this snippet for a line:
\n<!-- line primitive: -->\n<line x1="0" y1="80" x2="100" y2="20" />\n<!-- path equivalent: -->\n<path d="M0 80l100-60"/>\n
\nBut this was not the case with Medium's new symbol icon. Round shapes have to be represented by arcs and curves, and as you can see from the code above that always means more instructions for the drawing.
\nFor learning more about SVG, especially the path
element, I highly recommend to visit the compact tutorial at MDN.
Why do I even care? Sure the current symbol data is already pretty small and will not significantly add to the total payload size of a common Medium page, no matter if the article is long or short. It's not so much about the specific amount here, though it will add up to several gigabytes on their bandwidth bill eventually. I argue here only on a very high level and out of principle.*
\nOf course, the icon will be used in different places, different formats, different presentations. Not all of them will be SVG by default. I can imagine that the origin file is some Adobe Illustrator format or worse a Photoshop file. Probably I'm the very first who actually looked at the generated source code for the data in the HTML, as for many there is no real need to tamper with the bits and bytes besides running it through some kind of optimizer maybe. And for most parts in life that's okay.
\nThere is no need for over-/micro-optimizations unless you work in a field where this is a hard requirement.
\nBut here I thought it was not only a missed opportunity, but such a plain sloppiness. We're talking here about a company, not just a hobbyist's side project. There are people getting paid to do their job. Using simple shapes was so obvious after looking at the new logo. Am I the only one thinking that?
\nHey, Medium, if you want to use it, you are invited to copy my shorter version and apply it on your site! Free of charge, though some credit would be nice.
\n\n To your continued success,
\n SVG Wrangler & Byte Counter
Web technologies have come so far, that you realize: not everything needs to be done in JavaScript nowadays anymore.
\n\n\n\n»Life is really simple, but we insist on making it complicated.« — Confucius
\n
My initial headline would have been:
\nHow I wrote more JavaScript in the backend to eliminate JavaScript in the frontend.
\nBut that's already a mouthful and also would have revealed too much and you might not have clicked my slightly clickbait-y title, right?
\nSo here is my Public Service Announcement:
\n📣 This site does not use any frontend JavaScript.*
\n*) There are only tiny exceptions on 2 pages, but for a good reason. More on that later.
\nFirst of all I am not against JavaScript (JS) at all. If you're building a web application, then this is not only totally fine but most likely a core requirement.
\nBut I do have my pet peeve with JS for plain websites and blogs. Currently there is still such a strong draught in the static site generator world, telling us all our sites should be some kind of React or other frontend framework based project (looking at you, Gatsby, Next, Nuxt, VuePress, …). That you need to have that plentyful of code running in the browser of your visitors to have a smooth and feels like a native app user experience. That a site should be a Single Page Application (SPA). I can tell you, a plain HTML+CSS website does it really well, too. Surprise!
\nWhile on one hand the browser vendors add more and more Web APIs, we also got a lot of improvement in the HTML and CSS area. Usually there is no big hype train around them, unless you are very enthusiastic and live in that niche.
\nTake a look at caniuse.com to get an idea what is possible today and what might come tomorrow. Did you know that HTML5 is still iterated on and we're moving towards version 5.3? On the other hand »HTML 5« is also used as an umbrella term for a wide variety of standards. Also for CSS the story got very interesting: while CSS until 2.1 was a single specification, since CSS 3 there is a whole potpourri of recommendations and drafts. The wiki of the CSS Working Group might be a good starting point for further discovery.
\nBut I want to give you some more practical examples and an experience report:
\nThis is something you can observe here on this blog:
\n\n\nThe key ingredient is the CSS position: sticky
🛈. Even though most of them are labeled as partial support, this property value can be used in most scenarios except in some table related cases. If you want a sticky menu after scrolling and use only elements like div
everything is just fine. I could throw away all the code for that after I realized that none of the common and modern browsers had any blocking issues. So I did. The only real latecomers were the web view components, no big deal for me here.
const navbar = document.querySelector('.navbar');\nlet sticky = navbar.offsetTop;\nconst navbarScroll = () => {\n if (window.pageYOffset >= sticky) {\n navbar.classList.add('sticky')\n } else {\n navbar.classList.remove('sticky');\n }\n};\n\nwindow.onscroll = navbarScroll;\n
\n.navbar {\n position: relative;\n}\n.sticky {\n position: fixed;\n top: 0;\n left: 0;\n}\n
\n// nope\n
\n.navbar {\n position: sticky;\n top: 0; /* it does not reposition right away,\n but determines at which point it sticks */\n}\n
\nThe workaround with JS is no more. Yay!
\nAlso notice how little code is actually needed now? Two CSS properties and the job is done.
\nAlso in 2018 I played with Progressive Web Apps (PWA). The whole blog was one. A few days ago I teared down all of it. At the core of PWAs sit Service Workers (SW), though you can use SW also without building an app. And that's what I was aiming for, but in the end my home-grown dynamic cache solution was more annoying to me than helpful for anyone else. Every time I updated anything here, I had to wait and/or force refresh to see the result. I'm sure some people probably see visual inconsistencies due to a still running service worker in their browser. If you do, try to force clear all data for this website.
\nLong story short: if you do not build a web app, you most likely do not need service workers. So yet another thing down from the JS list.
\nNo before/after comparison here, but several precious kilobytes of JavaScript shaved off by removing them.
\nWoa, what are all these random acronyms here? Don't worry, the simple answer is:
\nIf you have images and they are not very small in file size, you maybe want to provide a temporary placeholder with very low resolution and quality. This is pretty useful for slow internet connections; living here in Germany I know how difficult this situation can be. That thing called internet is still very Neuland to us. 🤦
\nAnyway, SQIP
can be translated with »SVG
-based LQIP
.«
SVG
are Scalable Vector Graphics, an image format I really love a lot, my logo is done with it (I wrote about it a while ago).
LQIP finally stands for »Low Quality Image Placeholders« and is based on an algorithm to find primitive shapes to describe the source image. Basically try to find only a few triangles, rectangles, circles, ellipsis, and other low poly shapes. It is also an art form in its own, you can enjoy some nice examples there. The advantage of SVG is that it is made to encode such figures in very few characters of human readable text, so a less complex image for a placeholder can be written in one kilobyte or less.
\nCompared to the original high resolution image which can easily weigh half a megabyte and more this is great. You can reserve the space in your page and very early in the loading process display some visual hint that there will be a proper picture soon. Especially for types which do not support progessive loading (as JPEG can) using SQIP/LQIP placeholders makes a lot of sense.
\nIn this scenario at first it was not really about saving frontend JS, more about saving it on the backend site and replacing it with something else. Unfortunately in between some code creeped into the frontend anyway.
\nBut what happened that this beautiful technique fell out of favor with me?
\n<picture>
Enter another interesting HTML tag combo: <picture>
with <source>
.
So one reason to use small low quality placeholders is because before such tags became a thing we solely relied on a single <img>
and some trickery with CSS (and sometimes aided by sprinkles of JavaScript). I tried to avoid JS completely, but of course I had to use some styling hacks eventually.
The essence of it was some style attached to the image in question:
\n<img src="highres-and-heavy.png"\n style="background-size: cover;\n background-image: url('data:image/svg+xml;base64,PHN2…');">\n<!-- Usually in some post processing all style attributes were collected\n into a <style> tag or CSS file. -->\n
\nThe JavaScript entered this scenery at one point: after I used images with transparency.\nSadly with this background image workaround you would've seen the low quality placeholder through the transparent parts, and this was extremly ugly to be honest. I could not stand it and deployed some snippet to trigger a background removal once the actual image was loaded:
\n// remove the background image styling, so transparent images won't have\n// strange SQIP artefacts shining through\ndocument.querySelectorAll(\n "img[loading=lazy][class]:not(.thumbnail):not(.loaded)"\n).forEach((img) => {\n img.onload = (_event) => img.className = "loaded";\n});\ndocument.querySelectorAll(\n "img[loading=lazy].thumbnail:not(.loaded)"\n).forEach((img) => {\n img.onload = (_event) => img.className = "thumbnail loaded";\n});\n
\nTheoretically it would have been tolerable, but I noticed some strange behaviour once I started wrapping my images into picture
tags.
Let's shave the yak a bit further to understand why.
\nCome on, more acronyms? I'm sorry, the web is a place with a lot of them.
\nAll you need to know for now is that both of them are pretty modern image formats with quite good (lossy) compression rates while keeping a respectable quality. WEBP
has been around for some time and most of the browsers do support it. AVIF
is extremly new and right now only Chrome since version 85 and Opera 71 can display them. Firefox has a configuration flag, maybe they will enable it by default pretty soon.
So the current situation is that I have my original image (PNG or JPEG in most cases), a WEBP version, an AVIF version, and the SQIP placeholder. How do I deal with it? Back to our <picture>
tag:
<picture>\n <source srcset="./cover.avif" type="image/avif">\n <source srcset="./cover.webp" type="image/webp">\n <img src="./cover.png"\n style="/* SQIP data: see example above */">\n</picture>\n
\nYou can also use source sets for different view sizes based on media queries, but in my case I'm mainly concerned about supporting different image formats. My idea is to prioritize the formats with the smallest file size first, and given the compression ratios the order is usually: AVIF, WEBP, PNG/JPG. Not in every case will this be true; WEBP does not always have better savings then a decently compressed JPEG for example. AVIF has not disappointed so far, but sadly a part of my visitors will not see the effect yet.
\nWhat did not really happen anymore was a display of the placeholder before the final image was loaded. I experimented for quite some time until I realized that I do not want to spent more energy any further.
\n\nI made a risk-return tradeoff compromise and got rid of SQIP altogether. For the growing number of AVIF support the images are sometimes significantly smaller which makes it acceptable to allow for some display delay anyway.
\nIn the following screenshot the JPEG was the source photo. The PNG was created for some transparency stuff; of course, for photos this format does not really make a lot of sense in general. Sadly also WEBP fails to compete in this scenario. That's why I have to make this picture group generation a bit smarter soon to reorder based on the actual file sizes. Right now it is just based on a fixed format preference choice.
\nWell, AVIF is beating it definitely here. And yes, this 17.6 KB version is visually comparable to the original one. Sure, a closer look and you can spot differences, but for supportive symbolic pictures 100 % accuracy is not required. If you don't pay attention to it (or just don't know the source), it can compete with ease.
\n\nJust as an aside: during deployment another attempt of compression is done, the PNG gets shrunken down to 159 KB, the WEBP not at all.
\nThe only annoying part is still that the image dimensions are not reserved appropriately, even with present width
and height
attributes. So the layout can jump around if you scroll past the not yet loaded image. If you have an idea how to deal with that within picture
groups, I love to hear about it.
More images in different formats, more backend processing (and for now mostly in JS), no placeholder images, no frontend scripts.
\nYou can find my current attempt for »<img>
to <picture>
« transformation in the GitHub repository for this site.
\n\n»Sometimes I will add something here and there. And just a bit later I think I should remove something again.« — me
\n
Basically I removed all scripts not essential for visiting, viewing, and reading. Visually pleasing hints like a subtle progress bar for the reading position were a nice toy, though not necessary at all.
\nIn the beginning I mentioned, that there are exceptions to the JavaScript-free site. You'll find scripts for the pages resolution and theme.
\nThe first page is actually for myself when I need to check screen and window sizes while tweaking my theme. It needs to dynamically (re)calculate those values when resizing and scaling, so there is not really a way around JS here. And as said, I don't mind it at all.
\nThe second page could live without client side scripting if I'm honest, but back then I was a bit too lazy to automate the data generation in a fully static fashion before or during deployment. Nowadays it serves as a reminder that you can get the computed style programmatically. Sometimes this can be very useful.
\nBy the way, this site is powered by 153 MB of node_modules
during build, pre & post processing, and deployment times. I'm not immensily happy about it, but currently that's okay. Though I will probably work on replacing that with something else in the future.
Another side fact: since I do not track my visitors, not even any third party scripts are lingering around anywhere. This also means you have to not only bring coffee or tea for your afternoon reading, but also your very own cookies, too. Because I don't eatneed them.
I wish more people—specifically working or being interested in the web development field—would consider to reduce or avoid the JavaScript cruft in their projects. When you build a static site, also ask yourself, what "static" really means to you? (I remember the slightly darker sides of the era of DHTML, Java Applets, and Flash, brrrrr …)
\nCall me oldschool, but if you create a site with mostly text to read and images to view, why would you want to try to turn my device into a heater or stove? The simplicity of things also lies in what not do add and what to leave out.
\nI hope I gave you some ideas with the examples above. Discover the HTML and CSS specs and look for opportunities to replace parts with smaller and smarter solutions. If something is still rough around the edges, don't worry, most of the things are living standards and constantly evolve and improve over time (and you can help if you want to). Browser vendors usually implement some draft before the proposals and specifications are final. Everything is moving fast. Keep your eyes open!
\n\n\n»Enjoy the little things, for one day you may look back and realize they were the big things.« — Robert Brault
\n
And o hey,
\n🦄 ❤ JS
With WSL2 filesystem performance degraded for the mount points of the Windows host. Here's a tip to speed up git status
again.
tl;dr: jump to the solution.
\nFor personal projects I work on my PC. And to my own surprise nowadays solely on my Windows 10 installation. While I still have my Linux on another hardrive, I haven't booted into it for half a year now. The reason why it is possible, specifically if you do have Linux focused projects, is WSL
(Windows Subsystem for Linux). While I have experimented already with WSL 1 for some time, I switched to WSL 2 since it became generally available in Windows 10 Version 2004 (apparently it was backported to 1903 and 1909).
My main reason for switching to WSL2 was much better Linux support (kernel and syscall stuff), and for a long time everything was fine. I didn't even notice the performance hit across OS file systems.
\nBut lately something was bothering me. And it started with a related but independet issue in my favourite shell prompt tool starship: git_status
became extremely slow in repositories of some size. Since it was my prompt every command execution in a git-managed project was slow, better yet, it took just seconds until the prompt came back. First I didn't know what the issue was, until I discovered issue 1617 (and in another issue it is mentioned that an update to v0.45 will make it even slower, as well as a solution idea to mitigate it).
So I turned the extension off and ran git status
manually. But lo and behold: that was still very slow!
(click image for bigger version)
10 seconds (Linux git) vs ~400 ms (Windows git.exe)!
\nAside: hyperfine is a nice tool to benchmark CLIs and shell scripts/functions.
\nWhat was the problem? Also under Windows in a PowerShell with the Windows version of git, no performance penalty was noticable. I searched around, if there was an issue with git under WSL2.
\nAnd then I got reminded about the one line of the comparison table mentioned above:
\nFeature | WSL 1 | WSL 2 |
---|---|---|
⋮ | ⋮ | ⋮ |
Performance across OS file systems | ✅ | ❌ |
Well, the row alone does not really explain how much of a problem this might be.
\nI was not alone, in the GitHub repo of WSL I found issue 4401 which was quickly pointing to 4197. Within the lengthy thread there was a very detailed explanation of the underlying problem, some key statements:
\n\n\nWindows files are now accessed across the VM boundary, however. […] In WSL2, every operation has to send data to the host, exit the VM, wait for the host to perform the operation (which still involves emulating Linux behavior and the cost of Windows IO operations), send data back to the VM, trigger an interrupt in the VM, schedule the virtual processor to run, and continue executing in the VM.
\nEssentially, Windows files are now a "remote" file system.
\n
\n\nTo make matter worse, to ensure the same behavior as WSL1, we don't use any caching.
\n
\n\nWithout caching, it means every "stat" operation has to make a round-trip to the host (multiple, actually, partially because of how Linux VFS works, and partially because of how 9p works).
\n
Leaving with one recommendation in the end:
\n\n\nSo yes, for now the most important thing is: if it's at all possible, store your projects in the Linux file system in WSL2. This will be much faster than anything in WSL1, and gives you the full benefit of WSL2.
\n
Sadly that's not an option I want to follow, as I love to work on my project across OS boundaries. So I have to wait until a performance improvement will land in WSL2 hopefully.
\nSince git
is the only very noticable issue for me, I only need a fix for that for now. Luckily someone else had a great idea how to alleviate the problem (discovered through a tiny detour to someone else's dotfiles).
\n\nI came up with a dumb but useful workaround for this specific case. It works because windows git is faster against ntfs and I don't need any linux specific things for git at least.
\n
I took the snippet and adjusted it for my environments.
\nCall the Windows Git (git.exe) command when working in a /mnt
folder which is a Windows host (NTFS) mount.
Bash, ZSH, and friends
\n# use Windows' git when working under C:\\ drive\nfunction git() {\n if $(pwd -P | grep -q "^\\/mnt\\/c\\/*"); then\n git.exe "$@"\n else\n command git "$@"\n fi\n}\n
\nFish
\nfunction git --wraps git\n if pwd -P | grep -q "^\\/mnt\\/c\\/*"\n git.exe $argv\n else\n command git $argv\n end\nend\n
\nSome explanations:
\npwd -P\n
\n… returns the current physical working directory with all symlinks resolved. This is great since I have some symlinks for my dev folders, so all machines and VMs have a common structure (well, except for Windows itself).
\ngrep -q "^\\/mnt\\/c\\/*"\n
\n… searches for the pattern matching paths starting with /mnt/c
quietly (-q
), the exit status will be used for the if
check. Adjust the pattern if you need different drive letters.
git.exe "$@" # fish: git.exe $argv\n
\n… calls the git from Windows. This is one of the benefits of WSL over usual virtualization approaches: Windows allows to execute commands within WSL environments. IIRC it basically executes directly on the Windows host and only communicates back and forth. So git is called on the C:\\...
folder and no filesystem penalty has to be paid.
command git "$@" # fish: command git $argv\n
\nSince we're masking the git command with our function, we need a way to call the git executable directly. The linked solutions use explicit absolute file paths, but I don't want to think about if a distro installs it under /bin
, /usr/bin
, or elsewhere, I prefer to let the shell figure out where the executable is. command
is made for that. Reminder to myself: use command more often!
So, my starship prompt is not fixed yet, on the other hand I can live without the git status info. Theoretically I could also live without this nifty shell function, since I moved most of my repo management into my VS Code workflows. But sometimes it is nice to do some tasks in a plain shell.
\nAlso if you're interested in my current development environment on my PC, ping me on twitter and maybe I'll post about it here.
"},{"id":"https://markentier.tech/posts/2020/10/rust-2021/","url":"https://markentier.tech/posts/2020/10/rust-2021/","date_published":"2020-10-04T10:04:00Z","banner_image":"https://markentier.tech/posts/2020/10/rust-2021/cover.png","summary":"Collecting my thoughts around what I would want from Rust in 2021.","tags":["rust","2021","edition","embedded","nrf5280","nrf-rs","arduino","platformio","avr","attiny","atmega","esp32","esp8266","async"],"content_html":"Collecting my thoughts around what I would want from Rust in 2021.
\n\n\nIf you count the birthday of a programming language only after it hit its 1.0 (stable) release, then in 2021 Rust will turn 6 years old. Of course, the language has a much longer history.
\nIn 2006 Graydon Hoare started it as a personal project and in 2009 Mozilla sponsored it (and since 2010 it became an official thing). So first of all: Thank you both, Graydon and Mozilla, for bringing that quite awesome language into existence!
\nYes, admittedly it has a very steep learning curve. As someone who only plays with it every once in a while, has no long term projects yet to use it, and also considers themself as a slow thinker, I still appreciate all the things this language, ecosystem, and community has to give. There are still many things which will take me a long time to grasp and turn into a second nature, but as a mostly end user I feel already productive nevertheless.
\nAnd yet, we shall not stop improving. Learning, growing, developing.
\nSo what do I want from Rust in the next year and beyond?
\nThe short summary would be to have an Arduino or PlatformIO like ecosystem at one point. And I know that there is already a loooot of development going on. Just recently I signed up for some Embedded Rust training/workshops, and as a sponsor to the Knurling project I also get access to another opportunity called Knurling Sessions. (Psshht, if you become a sponsor, you also get immediate access to them.)
\n\nTools like probe-rs' cargo flash and cargo embed are a great foundation for some chip families and series like nRF52s, STM32s, and LPC8xx. The rust-analyzer (RA) is an incredibly powerful tool to bring you the IDE experience.
\nSo, while the tool landscape is growing and evolving, I still feel a bit lost. If you're a beginner in both Rust and Embedded, it will be probably overwhelming to figure out what devices and equipment to buy and how to go from there. Sure, the embedded book focuses on one particular hardware, but there are quite some holes and blind spots in the guide; being on a Windows system you do feel a bit left behind, something the Arduino folks have understood.
\nAlso speaking of Arduino comparison: Even if their IDE is not the best in the world, the combination of software and hardware makes it incredibly easy to get started and have very quick success moments. Install IDE, plug in the board, select the Blink sketch and upload to your device. Even as a slightly seasoned developer and person with many years of experience in computer and tech I love such low barrier entries a lot.
\nKeep in mind: after the newcomers had their first blinky moment, they still have to learn a lot about the language and the microcontroller.
\nThe folks at Ferrous Systems walk already in that direction (defmt is really nice btw), but I believe we still need that well integrated bundle. A dedicated development/starter board + predefined IDE with examples might be really necessary to get more people into Embedded with Rust. I would be fine if it is build on top of VS Code and maybe even the nrf5280-DK, at least for the first iteration, but we definitely need some kind of installer for all the tools you need to get it started. We need to make it as low friction-y as possible.
\n\nBy the way a board with the Arduino Uno form factor would be nice. I know, lately all the newer boards get some smaller shapes. Yet you will probably still find a wide ecosystem around this Uno board like extensions and shields and cases …
\nFurthermore I wait for more first class support of established microcontroller types. For people who worked with Arduino in the past, it would be great to see the AVR Rust ecosystem evolving and growing. I haven't looked at the latest quickstart guide, but I think it is still not very beginner friendly yet. AVR might not be an actively developed branch in the microcontroller landscape anymore, but there are tons of devices out there. I have probably accumulated dozens to hundreds of some ATtiny chips and ATmega boards (they can be extremely cheap when bought in bigger quantities). And sometimes, a tiny 8bit unit will just do. Sure, I can use them with C++ tooling, but do I really want to?
\nAnother corner is Espressif's ESP32 and ESP8266 (both chip series based on the Xtensa architecture). I think one reason these chips became so successful was also due to the integration into the Arduino ecosystem, at least that's how I got started with them. And PlatformIO has good support for them as well. So yet again, we need to build up the compiler support, but also make it easy to get started with them using Rust. Same as with AVR based microcontrollers, I have plenty of ESPs lying and flying around, all waiting to get some Rusty code onto them. Recently there was some progress and the quickstart guide was updated, but you still have to be very dedicated and love to compile the needed toolchain yourself. Also checkout the esp-rs GitHub organization for more crates and tools.
\nCompletely independent from embedded development is this point about a more guided and standardized way of how we should deal with errors in our libraries and applications. There are now plenty of crates to help and support us. But for me this also means: too much choice, so which one to pick? What approach will be more future proof? Some people in the list below have dug deeper into that topic though, so if you're interested then check them out.
\nThere has already formed an Error Handling Project Group.
\nCurrently I find it still very annoying to have this divide between tokio and async-std. Both are great projects, but since also library authors pretty much hardcoded their preferences, it is sometimes hard or impossible for users to mix and match. I'm all pro multiple executors and runtimes, there's probably even a need for more than one. But let's find good abstractions between them, so it's easier for me and others to use. And I don't want to have take sides.
\nAfter you get over the first hurdles of the language, it might become difficult to advance in a pace you find appropriate. To make this point short: there's now a lot of material for the curious and for beginners, there's also material for extremely enthusiastic and expert level people, but the middle ground still feels like a barren land. There are some great folks creating awesome content on YouTube. And I just wish to be more of it. After watching Jon Gjengset (»Crust of Rust« series) and Ryan Levick's stuff, I really want more of that. So, please, if you feel comfortable with writing or creating videos, help others like me to grow further.
\nThis is a very personal point, but I wish not everyone would think of blockchain and cryptocurrency stuff the first time they hear about Rust and where it's used. Also the job market seems to be quite dominated by this topic; if I wanted to consider a career change (going professional with Rust development) I don't really see good alternatives yet (though IoT/embedded does seem to be a viable option). Can we focus our energy elsewhere and/or more broadly? There are plenty of interesting topics, but I don't believe that "blockchaining" all of them is the right thing to do. To be honest I yet have to see a very good and reasonable project for blockchain technologies besides the hobby gambling with my Bitcoins and Ethereums.
\nI'm quite okay with the current development of the tooling so far. cargo
is probably one of the best I have encountered so far; at least for pure Rust projects there's not much more you could ask for, either it's built in, a cargo plugin away, or merely an alias for more convenience. I don't have need for deeper or better integration to other languages and their build systems.
I mentioned it already above, but to close with it again: rust analyzer is the greatest thing that could have happened to Rust developers. Please keep going and improving it! I usually have a quite high failure tolerance, but others might want way better stability. But I definitely do not want to live without it anymore. Oh, and if you're an as happy user as I am, please consider supporting the maintainers (open collective, github sponsor).
\nIf I could wish for something in 2021, then that rust-analyzer becomes a more official part of the Rust language. I don't know if it should be directly adopted or get some other support (somehow via the to be founded Rust foundation maybe?).
\nJust for myself and others here the collection of blog posts as they appeared so far on This Week in Rust (#356, #357, #358):
\nThe initial announcement and call for blog posts:
\nhttps://blog.rust-lang.org/2020/09/03/Planning-2021-Roadmap.html
\nℹ️ This is a link heavy post. Anything broken or not pointing to the right place? Send me a message via Twitter or open an issue on GitHub. Thank you!
"},{"id":"https://markentier.tech/posts/2020/09/beginnings/","url":"https://markentier.tech/posts/2020/09/beginnings/","date_published":"2020-09-09T00:00:00Z","banner_image":"https://markentier.tech/posts/2020/09/beginnings/cover.png","summary":"They say, change is painful. You must become vulnerable. I hate being vulnerable. And yet I have to expose myself to change something …","tags":["sabbatical","burnout","depression","future","career","mental health","work","life","beginnings","restart","plan","personal","care"],"content_html":"They say, change is painful. You must become vulnerable. I hate being vulnerable. And yet I have to expose myself to change something …
\n\nI have a weird and strange past. I have not finished school, no formal graduation nor degree. Drop out.\nLost hope, got depressed. I tried to kill myself; and—fortunately?—failed there as well. Crappy jobs and social system. Lots of wasted time of my life.
\nLater I tried to get my A level and some office administration in a dual vocational education and training, but also something I had let slide again. Circumstances were difficult and I was more concerned about our social system than actual education. When you have to apply for Schüler-BAföG, Kindergeld, Halbwaisenrente, and Hartz IV, you definitely learn something no school will tell you. I definitely did. But you don't get a degree in surviving the system.
\nThese two paragraphs cover basically 25 years of my life. There's a lot more I could tell, but those are different stories for another time.
\nGetting into jobbing again. Found my first long term gig and even mildly enjoyable over time, as I could shift into a more technical area there (it was about online marketing and I should have focussed on acquisition, but ended up helping users with WordPress plugin setup and such stuff). This was from 2008 to 2011.
\nDuring this time I not only tried to hone my tech skills in general, but also more seriously got into programming and found a new language: Ruby. Oh boy, I wish I had discovered that many years earlier. It was definitely more pleasant to work with than with PHP and Perl back then. I felt this could be something I could stick with and enjoy doing for longer. I mostly played with it, but I had no big plans yet.
\nSome day in summer 2011 a friend approached me.
\n»Hey, do you know by any chance someone who's a Flash Developer?«
\nYes, this was still a thing 10 years ago. But nope, I didn't know anyone.
\n»Nah, I'm sorry. But you wouldn't know by any chance a company looking for a Ruby Developer?«
\nIt was a shot in the dark. The worst what could happen is that the arrow comes back and hits me.
\n»Um, yeah, sure. We do. Wanna talk with our CTO about it?«
\nAnd so I finally stepped into the professional software development. I made it very clear from the beginning, that I had no experience and would probably lack a lot of knowledge and skills. But I was willing to learn and absorb everything. And did get the position. It was not for too long though, but this initial trigger was all I needed to settle somewhere I truly felt comfortable and competent enough.
\nAnd yes, once you have your foot in the door, it also becomes much easier to find a new job in tech, because you finally got your first records in your CV, so it's easier for recruiters and HR to consider you. But to be fair I got extremely lucky over the years and had always a good connection for the next company on my path. For the last 10 years it still wasn't too difficult for a Ruby engineer to get a new job, as we have quite some startups which have built their products and services with Rails. And I would expect it to be still the case for some more years. Ruby's dead my ass!
\nSo, yeah, almost a decade of professional tech work. I made a hobby my job. Something I enjoyed a lot in my spare time years ago became my work. And surely does it pay well, no doubt there. But I also noticed changes.
\nIn the early days of my tech career I was full of energy. I visited basically every Ruby user group meeting (I consider myself inventory now), I was an active part in the community, built and contributed to open source software. I found some refreshing breeze in the Elixir world, and given that many Rubyists took a look I was again surrounded by nice folks again.
\nSo I spent a lot of time at work and at home doing tech stuff. Also vacation was something I needed to get reminded of. And if you jump into an early phase of a startup you might totally forget about that completely. Who needs time off when it's so much fun already?
\nWell, in more recent years I could feel the impact of it. Slowly I lost drive, motivation, and energy. My contributions came basically to a halt around 2016. I still didn't take vacation more seriously. Mostly a week here and there, maybe 2 weeks around the holidays. If possible. I mean, how can I leave that important work for more than a few days? We need to build a product and the team was small. They need me. And I need them. My work became my life.
\nThere were only brief periods of slightly different interests. Though fiddling with electronics and microcontrollers is way too close to my work anyway. I tried to pursue fitness, and while hitting the gym and lifting the weights and getting gains is a pleasure, there were things that made it also annoying; obsession over body image and diets, eating disorders, … let's say the fitness industry is very good in hiding the dark side of itself.
\nI tried some short distance traveling, but never left Europe. It's okay, I'm not a globetrotter. There are places I'd like to see eventually, but there's no rush. I can be a little bit stereotypical here and leave this stuff for my pension time, can't I.
\nSo there was a slow and steady downhill … and then came 2020 around. While the beginning of the year looked promising, around March it hit us all. And the realization that everything has changed.
\nFeeling already gloomy and dark most of the time, having no joy in life but that little bit of work, and then add social distancing, work from home, and isolation to the mix. »Oh, but you are an introvert, you should be fine.« Well, it is not that simple. Of course, I can entertain myself for a long time, but I still need people and social interaction. The complete absence is frustrating. The virtual meetings a very, very light substitute. And the anxiety to be constantly surrounded by willfully bad or ignorant people trying their best to spread a dangerous virus is basically killing me at least psychologically and emotionally. I love the city for it's comfort of life, but the public space became a hostile environment to me. Taking a train or walking through discounters and malls are pure stress.
\nChronically depressed, burnt out, anxious, socially isolated. What a happy life!
\nAnd I haven't even talked about my uncertainties and insecurities around professional career growth. I think one of the trickiest pitfalls is, when you become so invested in your current job, that you try to seek every opportunity to stay there (it's also comfortable after all) and find ways to progress as much as possible. But the hard questions I have avoided so far: Can I still grow there? Am I just afraid to try something new? A new beginning? It's quite unusual for me anyway, almost 6 years at the same company. Inventory again. Between the founders and me are only 3 more people. And all of them in slightly better position than me anyway. I'm so bad in selling myself that I feel like the janitor of a school you'll find there for the last 25 years. In startup years I probably have reached that status I guess. Oh, have I forgotten to tell you about my impostor syndrom?
\nOkay, pseudo-objectively viewed I am a decent programmer, if you have a web based product or service I'm capable of pulling off something somehow. And once I touched or navigated through most of the codebase I have enough context and experience to feed of for the future. At least I can provide history and anecdotes, just in case you need some sarcastic and salty entertainer.
\nI'm also getting more comfortable in designing components and systems. I have enough ops experience to deploy and run the stuff myself, out of necessity in most startups. Running your monolith on physical bare metal servers colocated in a nearby datacenter up to microservices in the kubernetes orchestrated cloud, I have seen now my fair share of everything.
Due to circumstances I was the lead for two projects (we call them opportunities or initiatives, but in the end it's still all the same), the second one with significant product management aspects. Admittedly more than I actually anticipated in my role as an engineer, but most career ladders will expect that from you sooner or later anyway. Check.
\nOver the last year I've also built up a beautiful mentorship relationship with a coworker. I'm truly impressed by their progress, basically fresh into tech from a bootcamp and now an invaluable part of the team, and still so eager to learn more and grow further. It has been true pleasure. (Thank you for the opportunity! Helping you has also helped me quite a lot.)
\nPublic speaking was not my strong suite yet, but I gave a few talks in the company and in meetups, I had a conference workshop once (what a disaster!). I haven't really written about tech that much, probably mostly due to »who wants to read or hear my crap anyway?« Yeah, confidence is my middlename. Not.
\nTalk to me about a topic I love in private, and I'm all in. I will overwhelm you with all the stuff I know and can share. But ask me to do that in front of a crowd? If I don't have to, I'll pass.
\nAnd writing into the internet always feels weird. Back then when I had blogs with a comment function I was afraid about feedback, and the growing landscape of social media platforms confirmed my suspicion, people are bad. They write mean things, behave as if there was no human on the other side. I don't like it. And so I stopped writing publicly.
Last but not least, am I a thought leader? I'm probably not CTO material. I don't come up with great ideas before everyone else. Sometimes I have a good hunch about what will work and might have a future. I still remember how I wished tabbed browsing would be a thing before tabbed browsing was even a thing. If I just had been employed by a company developing a browser, I could have been renowned by now. Adam Stiles won that prize. (And to my defense: I was a teen.)
\nAnd while everyone is still probably figuring out how to migrate their legacy system to some microservice oriented kubernetes cluster farm I think that the focus should be more into the serverless and edge computing area. The signs are there, the vendors built up their portfolios, but compared to good old tech this is all still in its early days. But with my developer hat on I can say this is definitely more fun that fighting with gazillion of YAML files to please some cluster to run my workload. (Don't get me wrong, I believe that kubernetes has its use cases and containerization was a very important milestone, but it is not a means to the end, it was mereley a beginning.)\nI wrote an company internal post touching this topic and related to a project. The future will tell if I will be right about that development and thus how much of this so called thought leadership is in me.
If you remove the sarcasm and lack of confidence, not too shabby so far, right?
\nWrong! tells my mind. »You're a fraud. You just got lucky. You don't know shit.«\nI still apologize for all the crappy code I wrote. I didn't know better. (And still don't know. I'm just better at hiding it now.) I apologize for all the mistakes, misdecisions, and misplannings. I apologize for not being able to take compliments. I cannot celebrate success. I always fear something will just blow up the moment I take pride in my work.
\nJust a little over 2000 words just to tell you one thing: I am stuck.
\nMentally I spiraled so far down that I needed to pull the ripcord. Or the Andon cord for more dramatic effect. HALT EVERYTHING!
\nSo in June 2020 I talked to my Engineering Manager about a break. A sabbatical. 3 months. We agreed on a date. And here I am. From September to November I have time to sort my life. Rest. Recover. Refresh. Energize. Find purpose. Find motivation. Find lov… just kidding, we're not trying to overachieve here. But some kind of happiness in life would be still nice. I don't really know if I'm capable of feeling true joy again, but I can try.
\nThis was my first day:
\n\n\n\nStarting today I have now 3 months to figure out, how to find more purpose in life again. Even before the pandemic everything revolved around work for way too long. No drive and motivation for anything else. I hope this long break will help me to realign myself.\n \n
\n\n\nI cannot fully describe the situation, it is parts burnout and parts my mental issues. And this year has amplified some of them, introduced new, shifted others.
\nIn the end what I realized: I cannot go on like this forever. I need a break. I need to rest and recharge. Recover.
\nI need to force myself to think about something else than work. Which is hard. Lately most of my social interactions have been with coworkers. I started the day with them, I finished the day with them. They become basically friends and family, at least in my mind.
\nIt doesn't matter if it's real or not. Oh, and a fair bit of warning: the wrong companies will exploit this. If they sell you the "family" vibe, run.
\nSo my mind screws reality. I have been an idealist and dreamer anyway. But I have trapped myself. Cornered myself.\nAnd now I don't know how to get out of it. So I take a break. 3 months. Long enough to get bored, so my mind needs to escape its trap it has laid for itself. I need to break up the false routines. Don't let the depressive part take over. Never stop going. Face my demons.
\n
\n\nI'm at day 1.
\nMy fears are still: Will they miss me? Will I have a job when I come back? What do they think of me? Will this hurt my career?
\n
\nIt's day one. I'm broken. I'm afraid.
\n
\nI'm crying.
\n
\nIt's a step.
\n \n
And now my first week is already over. I have done nothing. Zero. Just lots of Netflix, YouTube, gaming, sleeping, resting, existing.\nI have still checked work mails and peeked into Slack a few times, but it got better. I still feel guilty, about everything. About doing nothing, about checking stuff, about thinking to check stuff.
\nIt's like being an addict. And maybe it became an addiction. If your life is really fucked up then work can be a drug. I had smoked cigarettes, I know how much it is just the mind and less the body.
\nQuite saturated by all this passive consumption now, my fingers are itching to do something. That I can write these lines here, emotionally a bit more stable, is already a win.
\nI still have no plan how I want to spent the remaining time. (I'm probably not going to travel, infection numbers are raising again, and I also don't want to sit in a train or plane right now.)
\nThis piece has so many hidden topics I could dive in. Maybe this will be part of the journey. Maybe I have to reflect more on my past me, before I can really focus on future me. And maybe I should care less about if someone wants, needs, has to to read it. In the early 2000s I had less problems with that, so maybe I can do it again.
\nI need to come clean with myself, I need to become vulnerable, honest and true.
\nTo quote a fellow who's in a similar situation like me:
\n\n\nBut the exciting question still remains:
\nWho are you ultimately without your work?
\nChallenging and valuable at the same time if you ask me.
\n
This is definitely not an end. It is the beginning of something new.\nI don't feel that excitement, that spirit of optimism yet. And maybe I never will. And that's okay, too.
\nI just need to keep going. Somewhere. Anywhere.
\nThe writing is as chaotic as my mind. If you couldn't follow, don't worry. I do not expect this to make any sense. By writing down my thoughts and feelings I enter a cathartic space. This is my healing process, and you are invited to listen and watch. Be my guest, have a cup of tea or coffee. Wanna have some cookies?
\nThis is also the first post touching the human side of this blog, the part I was always missing here so far. I kind of feel that I should have started with something like this. And weirdly enough it had to be about me to realize this. 🤷🏻♂️
\n"},{"id":"https://markentier.tech/posts/2018/12/ruby-bundler-nokogiri-macos/","url":"https://markentier.tech/posts/2018/12/ruby-bundler-nokogiri-macos/","date_published":"2018-12-30T00:00:00Z","banner_image":"https://markentier.tech/posts/2018/12/ruby-bundler-nokogiri-macos/cover.png","summary":"Quick answer:bundle config build.nokogiri --use-system-libraries && bundle install","tags":["ruby","bundler","nokogiri","macos","osx","rubygems","gem","error","undefined","symbol","iconv","libxml2","x86_64","clang","鋸","rails"],"content_html":"Quick answer:bundle config build.nokogiri --use-system-libraries && bundle install
Every other year or so I want (or need) to install dependencies for a Ruby application on my Macbook, directly on the host instead of a VM or container. Mostly it's a Rails app.
\nAnd every time our most "loved" dependency bails on me: nokogiri
. I think it always fails on a Mac. (At least once.)
Because I never go directly to the documentation of whatever refuses to work, I usually google my way to a solution.\nIn my case this was then harder than it should have been, so I write this down here for me as a reminder.
\nThe next time I google it, I hope to find my own blog post and will make the same expression at the end. Again.
\nTry to get and build the dependencies:
\n# maybe a fancy Rails application\nbundle install\n
\nAnd after a while …
\n# snippet\nAn error occurred while installing nokogiri (1.8.5), and Bundler cannot continue.\nMake sure that `gem install nokogiri -v '1.8.5' --source 'https://rubygems.org/'` succeeds before bundling.\n\nIn Gemfile:\n rails was resolved to 5.2.1.1, which depends on\n actioncable was resolved to 5.2.1.1, which depends on\n actionpack was resolved to 5.2.1.1, which depends on\n actionview was resolved to 5.2.1.1, which depends on\n rails-dom-testing was resolved to 2.0.3, which depends on\n nokogiri\n# /snippet\n
\nNow if you run what is suggested …
\ngem install nokogiri -v '1.8.5'\n
\nBuilding native extensions. This could take a while...\nERROR: Error installing nokogiri:\n ERROR: Failed to build gem native extension.\n\n# ... snip ...\n\nUndefined symbols for architecture x86_64:\n "_iconv", referenced from:\n _main in conftest-451598.o\n "_iconv_open", referenced from:\n _main in conftest-451598.o\nld: symbol(s) not found for architecture x86_64\nclang: error: linker command failed with exit code 1 (use -v to see invocation)\nchecked program was:\n/* begin */\n 1: #include "ruby.h"\n 2:\n 3: #include <stdlib.h>\n 4: #include <iconv.h>\n 5:\n 6: int main(void)\n 7: {\n 8: iconv_t cd = iconv_open("", "");\n 9: iconv(cd, NULL, NULL, NULL, NULL);\n10: return EXIT_SUCCESS;\n11: }\n/* end */\n
\nYou can also check the logs for later reference, too.
\n/Users/chris/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/extensions/x86_64-darwin-17/2.5.0-static/nokogiri-1.8.5/gem_make.out\n/Users/chris/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/extensions/x86_64-darwin-17/2.5.0-static/nokogiri-1.8.5/mkmf.log\n
\nThis is a problem that it cannot work with the iconv library currently present for linking. This can either be due to what was shipped with nokogiri or you have installed a different version via homebrew.
\nAlternatively also another library can cause some troubles: libxml2
Then the output might look like …
\nRunning 'compile' for libxml2 2.9.8... ERROR, review '/Users/chris/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/ext/nokogiri/tmp/x86_64-apple-darwin17.7.0/ports/libxml2/2.9.8/compile.log' to see what happened. Last lines are:\n========================================================================\n _parseAndPrintFile in xmllint.o\n "_xmlXPathEval", referenced from:\n _doXPathQuery in xmllint.o\n "_xmlXPathFreeContext", referenced from:\n _doXPathQuery in xmllint.o\n "_xmlXPathFreeObject", referenced from:\n _doXPathQuery in xmllint.o\n "_xmlXPathIsInf", referenced from:\n _doXPathDump in xmllint.o\n "_xmlXPathIsNaN", referenced from:\n _doXPathDump in xmllint.o\n "_xmlXPathNewContext", referenced from:\n _doXPathQuery in xmllint.o\n "_xmlXPathOrderDocElems", referenced from:\n _parseAndPrintFile in xmllint.o\nld: symbol(s) not found for architecture x86_64\nclang: error: linker command failed with exit code 1 (use -v to see invocation)\n
\nWell, if one would pay attention to the output, sometimes the output already tells you what could be done.
\nFor the libxml case:
\nIMPORTANT NOTICE:\n\nBuilding Nokogiri with a packaged version of libxml2-2.9.8\nwith the following patches applied:\n\t- 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch\n\t- 0002-Fix-nullptr-deref-with-XPath-logic-ops.patch\n\t- 0003-Fix-infinite-loop-in-LZMA-decompression.patch\n\nTeam Nokogiri will keep on doing their best to provide security\nupdates in a timely manner, but if this is a concern for you and want\nto use the system library instead; abort this installation process and\nreinstall nokogiri as follows:\n\n gem install nokogiri -- --use-system-libraries\n [--with-xml2-config=/path/to/xml2-config]\n [--with-xslt-config=/path/to/xslt-config]\n\nIf you are using Bundler, tell it to use the option:\n\n bundle config build.nokogiri --use-system-libraries\n bundle install\n\nNote, however, that nokogiri is not fully compatible with arbitrary\nversions of libxml2 provided by OS/package vendors.\n
\nSo, the last commands are the things you should consider for a bundler based project.
\nbundle config build.nokogiri --use-system-libraries\nbundle install\n
\nYou can check the config:
\nbundle config\n
\nSettings are listed in order of priority. The top value will be used.\ngem.test\nSet for the current user (/Users/chris/.bundle/config): "rspec"\n\ngem.mit\nSet for the current user (/Users/chris/.bundle/config): true\n\ngem.coc\nSet for the current user (/Users/chris/.bundle/config): true\n\nbuild.nokogiri\nSet for the current user (/Users/chris/.bundle/config): "--use-system-libraries"\n
\nThe file ~/.bundle/config
for completion:
---\nBUNDLE_GEM__TEST: "rspec"\nBUNDLE_GEM__MIT: "true"\nBUNDLE_GEM__COC: "true"\nBUNDLE_BUILD__NOKOGIRI: "--use-system-libraries"\n
\nAnd that's it. Fixed!
\nIf I had consulted the nokogiri documentation at all, I also would have got the hint earlier:
\n\n\nNokogiri will refuse to build against certain versions of libxml2, libxslt supplied with your operating system, and certain versions will cause mysterious problems. The compile scripts will warn you if you try to do this.
\n
It focuses on libxml2, but the steps also help with the libiconv issue, too.
\n🤦
"},{"id":"https://markentier.tech/posts/2018/05/minimalism-focus-clean-redesign/","url":"https://markentier.tech/posts/2018/05/minimalism-focus-clean-redesign/","date_published":"2018-05-01T00:00:00Z","banner_image":"https://markentier.tech/posts/2018/05/minimalism-focus-clean-redesign/cover.png","summary":"Start with whatever you have, but don't stop reaching for something better. Why I love minimalistic and simplistic web designs.","tags":["minimalism","focus","clean","redesign","design","webdesign","svg","scalable vector graphics","vector","graphics","simple"],"content_html":"Start with whatever you have, but don't stop reaching for something better. Why I love minimalistic and simplistic web designs.
\nA few days ago I pointed a colleague to my freshly hatched blog, because I wanted to show him my recent post. In the end we didn't even chat about the post, but web design and the job (he's a web designer), challenges and opportunities and interesting stuff to do and creativity and the past and the kind reminder that I'm not the youngest anymore. What kept nagging in the back was the comment about my (now past) design.
\n\n\n\nOh man you need need need to update your site
\nWant some design help on that bitch?
\n
As much as I appreciate the help and support, but especially for my personal project I tend to do as much as possible on my own. Also taste is highly subjective; I live in Berlin, Germany, I see stuff, I know what I'm talking about. And I do not need to agree with the style and display of others and others' work. I nod (or shrug), I go, and probably forget about it ten minutes later anyway.
\nBut there was a point to the above message. Something he couldn't even know. The design was a temporary solution, and I dug it up from a 4 years old proof of concept work I've done. It was the quickest way to get my site up and running without looking too shabby. I didn't need to seek for a pre-made design somewhere else and I didn't need to spend too much time on creating one from scratch.
\nWhile an old design doesn't need to be bad at all, mine wasn't really finished and fully thought through though. And so this sparked the urge to do something about it.
\nI also knew that I wanted to change the logo independently from a redesign, but now all things came together.
\n\nI love minimalism, I really do. And I wish I could follow this principle a bit more in my daily life, for now the digital one needs to be sufficient enough for this.
\nNot all my previous designs and themes were minimal, simple, or even easy to use. Some were artistic; I think in the early 2000s artsy, graphical and very detailed themes were a thing. Sadly I forgot to bookmark (and screenshot) some of my favourites, I cannot even remember the names of the people, so no hint how to find them again. Well, it's history anyway.
\nIn the end I always come to enjoy a design and layout which is not overloaded with stuff and cruft. Sometimes I will add something here and there. And just a bit later I think I should remove something again.
\nAlso: minimalism doesn't necessarily mean total emptyness or absense of everthing. But I came to love the huge blanks, and the vast endlessness of the plains.
\n<!-- basic SVG logo markup for markentier.tech -->\n<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1024 1024">\n <path fill="none"\n stroke="#0c1328"\n stroke-linecap="round"\n stroke-linejoin="round"\n stroke-width="64"\n d="M480 506v208-128s0-72 64-72 64 72 64 {…}"/>\n</svg>\n
\nI love vector graphics. And so I love SVG. I remember to have always liked this image format since discovery. It was quite sad, that it also took quite some time to have proper cross browser support. If you ignore old IEs, then the present day situation is awesome. SVG v2 is not there yet (and I really want to get rid of the xlink
namespace), but the currently implemented spec is already good for a lot of use cases.
One perfect example are logos and icons. As you already see, my website's logo is delivered and rendered as SVG, I couldn't get it smaller than that anyway, any PNG, GIF or JPEG would always be much, much bigger than the following path definition:
\n\n\nM480 506v208-128s0-72 64-72 64 72 64 72v128-128s0-72 64-72 64 72 64 72v128-128s0-72 64-72 64 72 64 72 1 85 1 128c-1 28-24 54-24 54-126 161-348 207-527 111-179-97-262-309-196-501 66-193 261-309 462-275 200 33 347 206 348 409
\n
Yes, exactly, this is all. It is probably the most compact version I could use. Hence the usage of a 1024x1024 viewBox and big numbers, just to avoid decimals, which carry the extra dot. This is a pro-tipp btw, always design your stuff on a huge canvas 1000 pixels or bigger in dimensions. Then with tools like SVGOMG (or already in your vector graphics program) you can save the file with zero decimals (so only integers) without loosing significant information.
\nIf you want to get more excited about SVGs:
\n\n\n\n\nThere are definitely more, but above videos should be a nice starting point. I might come back to this topic in the future.
\nThe screenshot above might look a bit weird, because it fits perfectly into the current design (I added a border so it doesn't look too floaty).
\nI think this represents an interpretation of minimalism. At least one I like. There didn't went too much process or sciene into it, at least not consciously. Of course, I'm aware of some color basics, contrasts and a basic understanding of accessibility. It won't be perfect yet. And probably I won't bake in some changes you'd like to see.
\nBut I do believe that my articles should be consumable without too much distraction or fatigue or strain. Since it is heavily color range reduced, color blind people (or with partial deficiency) should be able to read and navigate the site without loosing anything or getting confused. I haven't tested screenreaders, but I hope I don't use design magic which would impact your experience here.
\nI also still work on the performance aspect of it. And I also push for more modern browsers. Since it is my personal blog, I think I can afford to "loose" some customers, or at least give them not the full pleasure by not being always fully backwards compatible.
\nIf you have questions or just want to know in more detail how something works here, ping me on twitter or open an issue at the repo for this site, I'm happy to explain, glad if you can learn something, and I welcome feedback and corrections.
"},{"id":"https://markentier.tech/posts/2018/04/rebeccapurple/","url":"https://markentier.tech/posts/2018/04/rebeccapurple/","date_published":"2018-04-11T00:00:00Z","banner_image":"https://markentier.tech/posts/2018/04/rebeccapurple/cover.png","summary":"While polishing my theme, I discovered Rebecca. I love it, when technology gets some human touch.","tags":["rebecca","purple","color","css","eric meyer","honor","cancer","til","today I learned","tiny bits"],"content_html":"While polishing my theme, I discovered Rebecca. I love it, when technology gets some human touch.
\n\nI was just reading about the color data type in CSS over at MDN when I saw that a new named color will be part of the CSS Color Module Level 4.
\nThere was a link to this Codepen blog post.
\nRebecca was Eric Meyer's daughter. She was diagnosed with cancer. On her sixth birthday she left us.
\n\n\nThis past Friday, June 19th, 2014, the community to which Eric has given so very much has seen fit to name #663399 as rebeccapurple with his permission.
\n
Eric Meyer is probably most well-known for reset.css.
\nMore TIL: Btw the changes for CSS 3 to 4 are also interesting from a technical perspective: https://drafts.csswg.org/css-color/#changes.
"},{"id":"https://markentier.tech/posts/2018/04/hitchhikers-guide-to-structured-data/","url":"https://markentier.tech/posts/2018/04/hitchhikers-guide-to-structured-data/","date_published":"2018-04-03T00:00:00Z","banner_image":"https://markentier.tech/posts/2018/04/hitchhikers-guide-to-structured-data/cover.png","summary":"A tiny journey to the building blocks of the semantic web and how they make our lives much easier.","tags":["structured data","wikidata","contentful","extension","douglas adams","hitchhiker","guide","galaxy","semantic web","rfc1118"],"content_html":"A tiny journey to the building blocks of the semantic web and how they make our lives much easier.
\n\n“There is a theory which states that if ever anyone discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened.”\n \n
When I started using computers, I never thought about if and how they could help me to do things — and if that in turn would improve my life. I automatically assumed that we invented tools, machines, devices, phones, computers, and so on because we wanted to make our lives better, and to think less about tedious tasks by making the complex and especially complicated ones easier and more usable.
\nIn the late 90s I not only learned but experienced that computers are connected. There was this thing called internet, mostly advertised in the form of WWW, the world wide web: gazillions of documents more or less linked to each other. Mostly plastered with flashy animations saying that this place is under construction. And the <marquee>
tag made us feel as important as breaking news or a stock market ticker.
Ah, yes, the good old internet. In the beginning quite static and mostly text, but it became richer, more versatile, dynamic, and included images, audio, and video. Besides the static sites, we built web apps, APIs and made the internet a place to interact with each other and with machines.
\nAs a kid I read tons of books, mostly fiction, of course, but I remember I visited our library in the village quite often and borrowed books explaining the world to me. But I consumed so much, I couldn't keep all the information in my head, nowadays I'm not significantly more knowledgeable than decades ago. I remember bits and pieces, but more importantly, I usually remember where I need to look for the actual data to fill in the gaps.
\n\nI grew up with Google from the beginning. And while the old and manually maintained catalogs and directories were a good starting point to explore the digital universe, search engines made my life easier. Now I didn't even need to keep an index and graph of information fragments in my head anymore, I could shift to the skill of asking the right questions.
\nBut there are a few problems with most general purpose search engines. The easiest way to build an index is to crawl the internet and grab the full text of every page discovered and connect it to keywords. Later when somebody searches for this keyword they will get a huge list of results. The algorithms became smarter, the better you can formulate your query, the more relevant and condensed the list will be.
\nBut if you think about it, you can already ask very specific questions, and most search engines wouldn't really be able "to get it" right from the beginning.
\nIf you would have tried this maybe 10 or even just 5 years ago and you got lucky, then some result on page one would have pointed you to an article about his publications. If you're not using Google or WolframAlpha the situation is still quite disappointing.
\nToday I'm quite impressed how this works out:
\n\nAs you can also see, the first organic result is about the author on Wikipedia, only the second one is close to what I was asking. But the top bar with the book covers is rocking it! (I first tried with Google.de and there it was just the 3rd result.)
\nAnd I will make a bold statement here: giving you such specific and contextually relevant information is not possible solely based on some full text search index; I believe this was driven by structured data (and potentially stored in a graph database … maybe).
\nMy internal QuickCheck program narrowed it down to "douglas adams books", but this is not the point (although parsing the input of a search box is probably a nice adventure on its own).
\nWe can further decompose our query and figure that it consists of a tuple: "douglas adams" and "books". The first one describes the name of a person, the second one describes an entity type, and obviously we want to find a connection between this very person and the books he wrote. Since people rarely mentioned which books they have read, the query parser can already make an educated guess here and put as a relationship type a "has written" (also "has published" could work).
\nOr expressed in a pseudo query language and very simplistic:
\nSELECT * FROM books WHERE author = "Douglas Adams"\n
\nOf course, first you would need to resolve an actual author/person entity from your database:
\nSELECT id FROM people WHERE name = "Douglas Adams"\n
\nAnd then you realize that your people table/database could have multiple entries, and you would try to filter based on occupation or hobby. Maybe you have them ranked by popularity and will just take the highest ranked entry. Another option is to resolve the full query for each person and check which one has written the most books (if any at all) and use this one.
\nWhatever the approach is, you need to have your data stored in a structured way. And if you want to build a search engine, a knowledge database, or any kind of application querying and presenting such information (but not storing all the information itself) you need to rely on other sources providing them in a machine consumable way.
\n\nThe idea of structured data and a semantic web is not new. And if you have used any blogging software then you also used some aspects of structuring: categories and tags.
\nDepending on your needs you even introduced your own taxonomies to describe and differentiate your articles. As an example: you're a movie reviewer and classify your reviews based on mood, target audience, and genres; all of these dimensions help you to describe your data, make it more discoverable, support consuming systems, and directly or indirectly give your readers more context and meta data, usually in a quicker and easier way, so they can decide if they want to dive deeper into your blog or look somewhere else.
\nThe article previews/cards on Facebook and Twitter are also driven by structured data. When you look into the source code of some HTML pages you can spot some interesting meta tags with names or properties like "og:title"
, "twitter:creator"
, or "article:author"
. They are used to fill the pieces of the preview. And to be honest — you will most likely click on a blog post when it has a nice cover image and description line won't you?
It took us years to bring in more data, or to take advantage of the full potential of structured data, but nowadays this concept is the foundation of the web. Even with AI on the rise machines can do their job best if they can process the data in a structured way.
\nI think the art is to display content in a pleasant, human-readable manner and unobtrusively enrich it with machine consumable semantic information. This so that we can build interfaces and applications that can make sense of all of the data, and then visualize it in a different and more approachable way.
\nIf you want to see what structured data can look like, then head over to wikidata:
\n\nThese entity pages are very formal and not the most beautiful ones to read about a person's life, but they contain very valuable information by describing what this entity is about and what kind of relationship it has to other entities.
\nSuch data is designed to be consumed by other services. For example the infoboxes on the right side of Wikipedia articles are mostly supplemented with wikidata information.
\nSome people have built tools to nicely visualize such data:
\n\nYou have built a website for your international book club. It comes with a knowledge base, which supports already 42 languages. While managing the content for the authors and books is pretty easy with Contentful, you realize that localization is sometimes painful, because you have a hard time to figure out what the translations of each person's name are.
\nA friend of yours comes up with the idea to create a UI extension for Contentful, where you can look up the wikidata and automatically fill the translation fields for the names and other common properties. She has even developed a first prototype for you:
\n\ngithub.com/asaaki/cf-wikidata-ui-extension-example
\nYou really like it and start to think about how to automate a lot of other steps. For example the extension could fetch all the known books and create drafts for each of them and link them back to the author entry. Since wikidata also provides links to other Wikimedia services you could store them with your knowledge base articles as well. You integrate a lot of links to other sites and maybe want to create some kind of preview box similar to Twitter Cards.
\nWith well maintained structured data the possibilities are unlimited. So, go out, explore, and build awesome stuff.
\n\n\n\n“Curiously enough, the only thing that went through the mind of the bowl of petunias as it fell was Oh no, not again. Many people have speculated that if we knew exactly why the bowl of petunias had thought that we would know a lot more about the nature of the Universe than we do now.”
\n
\n—Hitchhiker's Guide To The Galaxy
\n\n“Curiously enough, the only thing that went through the mind of the bowl of petunias as it fell was Oh no, not again. Many people have speculated that if we knew exactly why the bowl of petunias had thought that we would know a lot more about the nature of the Universe than we do now.”\n \n
So Long, and Thanks for All the Fish! 🐟
"},{"id":"https://markentier.tech/posts/2018/04/progressive-web-app/","url":"https://markentier.tech/posts/2018/04/progressive-web-app/","date_published":"2018-04-01T00:00:00Z","banner_image":"https://markentier.tech/posts/2018/04/progressive-web-app/cover.png","summary":"This site is a Progressive Web App now, too. What does it mean and why are PWAs so cool? Main reason: it is about speed!","tags":["progressive","web","application","pwa","service worker","offline","caching","performance","speed","optimization","fast","google lighthouse"],"content_html":"This site is a Progressive Web App now, too. What does it mean and why are PWAs so cool? Main reason: it is about speed!
\n\n\n\nProgressive Web Apps (PWAs) are web applications that are regular web pages or websites, but can appear to the user like traditional applications or native mobile applications. The application type attempts to combine features offered by most modern browsers with the benefits of a mobile experience.\n \n
I haven't really followed the development of Progressive Web Applications (short: PWA
) for quite some time. And I think the last time I experimented with anything around this topic was, when Application Cache (or AppCache) was still a thing. So yeah, long time ago.
I have forgotten about web site/app optimization, simply because I had no real web site to build and maintain for years. The sites I had were either abandoned or I didn't care so much about the performance of them.
\nThe very first question a lot of people might ask is:
\nWell, if you do care about web performance and speed, you might want to consider all the tools available to you.
\nIf you in full control of the output (HTML, CSS, JS, images, fonts, ...) you have already lots of opportunities.\nYou can minimize and compress the files. The HTML, CSS, and JavaScript stuff is fairly easy, for all of them exist online services or CLI executables.
\nBut do not forget your assets like images (and audio/video if you host them yourself), most of them can be made smaller. PNG and JPEG are still the most used image file formats (well, also GIFs for animations), WebP is there, but I haven't really seen serious usage yet. Each format usually supports either lossless and/or lossy compressions. I still prefer PNG as my standard image format, and I like that there are tools out there helping me to squeeze out the last tiny bit to get to a pretty small file size.
\nBefore I come to the PWA part, let's have a look at what can (and should) be done, no matter what kind of website you have.
\nSo far I will only mention my go-to tool here: tidy-html5. I elaborated a bit further in my last post, when I wrote about changing my static site generator.
\nOn the other hand the overhead of bloated and whitespace cluttered HTML pages is proably not the biggest issue when it comes to fat sites.
\nInstead of recommending a specific tool I'd rather point to projects like webpack, parcel, or your favourite static site generator of choice with proper support. Also if you use a service like Netlify, that you can enable asset optimization and they will take care of the byte crunching for your, the same goes for the
\nEven if you use WordPress I know that people have written nice plugins to help you with such task. Nobody needs to do this by hand anymore.\nThe same goes for images as well, but for reasons I dedicate it its own section:
\nWhile services like Netlify claim to support compression of images, I'm pretty sure they will have trade-offs to publish and deliver your site faster.\nSince images are one of the biggest pieces of the total page weight, I think they deserve more attention beforehand.
\nFirst of all do you know which size you need already? Do not simply put your desired image unaltered into your blog post. Unless you always choose the too small images, you probably end up embedding way too huge files instead. Nobody really needs that 3000x5000 photograph when reading your article. If you want to provide them that original file for later usage, give it a link to it for download. So resize the image to your needs. Consider all the usual presentation layers. Most people can live with a 1024 pixels maximum on either height or width, depending on your theme you might be able to use even smaller default sizes.
\nI use an aspect ratio of 2:1
, so my default image dimension is 1024x512
, which is used for my cover and further images. I use this maximum as default even though my article body width is a bit smaller. The reason is that the cover image is also used for Twitter Cards (and Facebook) presentation. Both platforms either support or suggest bigger sizes, but for now I think this is overkill.
As said, I love PNGs, even though for photos the JPEG format might be better suited.\nAnyway there are a bunch of tools for PNG optimization.
\nThe most well-known is ImageMagick, which can be used for a lot of different image formats. I used it especially for resizing and generating my thumbnails of the cover images. Resizing down to the visibly rendered size saves already tons of bytes and therefore bandwidth!
\nSpeaking of thumbnails I go even further, because they cannot be only small in dimensions, but highly reduced and compressed, since they serve as a visual hook, but do not need to be the prettiest pictures ever.
\nSo something neat is the reduction of the color palette with pngquant; people familiar with GIF optimization now that this helps already a lot.
\nSecond is done by optipng. pngquant already can do a lot of heavy lifting, so maybe the companion doesn't so much afterwards, but maybe it will find a file for further optimization.
\nInitially I also had pngcrush in the pipeline, but after checking the actual byte size of the files I saw that it mostly increased afterwards; I removed it, because I'm not interested in de-optimization of my content. 😅
\nFor GUI lovers there is a handy tool for macOS users: ImageOptim. This works on a bunch of image file formats and with quite some tools under the hood.
\nMinimization and compression are just the beginning of a fast served site. Once all the files are up on a server, waiting to be delivered all around the world, they also do not change that often. Especially your very static pages with all the assets are unlikely to change (like blog posts and about and imprint pages).
\nSo, to prevent a constant flow of all the bits and bytes for each request, we want to cache them, and ideally as close to the user as possible, so that the next time they request the exact same data, they are served from their devices instead. This becomes highly efficient for repetitively used and requested files like CSS and JS, which are shared across your whole site. Also your logo and header images do not need to travel each time that way.
\nIntermediate steps are server-side caches and CDNs by the way, but I'm not focussing on this right now.
\nAlso: Caching in itself is a tricky story and should probably deserve its own article.
\nGenerally speaking the Cache-Control header is a good starting point and helps browsers to determine, what and for how long it should try to keep a file in at hand instead of fetching it from the internet.
\n\nAlthough I have heard of Progressive Web Application already, I never paid much attention to it. I surely thought that it might be cool to look into this topic, once I figured out a web app project. A static site was definitely not a trigger. I could not have been more wrong.
\nWhen I run the Audits tool in the Google Chrome developer tools, I had no idea what this Lighthouse stuff is. I remember that in previous version of Chrome, there was this PageSpeed Insights tab, which is deprecated and only available as a web service nowadays. It's still a useful tool on its own. But Lighthouse provides much more performance and optimization data.
\nSo when the report was generated I was a bit confused that it had this Progressive Web App section. Sure, some sites might need this information, but the whole internet is not only about dynamic apps, is it?
\nWell, it was about time to dive into this previously neglected topic again. I read and learned about the Web App Manifest (manifest.json) and the PWA Checklist.
\nOne important step for the baseline was:
\n\n\nAll app URLs load while offline
\n
With the suggestion to fix it by doing this:
\n\n\nUse a Service Worker.
\n
Oh, I also read about them, but for the same reason above I just ignored this topic, too. If you want to learn what you can do with them, check out serviceworke.rs
\nAs a starter I went for the pretty simple setup provided by Google. But also iterated further. Tell me if something doesn't work out well or the site just looks ugly (unstyled).
\nThe cool thing is, that it not only makes your site perform faster and cache items, but also provides offline functionality.
\n\nThe screenshots above are an example of how it would look like, if you decided to add markentier.tech to your homescreen.\nColoring the address bar is also a nice gimmick.
\nUntil now I cannot tell you much more about PWAs yet, since I'm learning the stuff right away on the go.
\nI encourage you to look into this topic as well. I hope I provided you with enough links to start your own journey.
\nHappy Easter! 🗿🥚🐰
\nPS: Sorry, no April Fools' Jokes or funny anything at all here.
"},{"id":"https://markentier.tech/posts/2018/03/from-cobalt-to-zola/","url":"https://markentier.tech/posts/2018/03/from-cobalt-to-zola/","date_published":"2018-03-28T00:00:00Z","banner_image":"https://markentier.tech/posts/2018/03/from-cobalt-to-zola/cover.png","summary":"I wish I could have the features of both tools, but for now I will use zola over cobalt. A tiny migration and feature comparison story.","tags":["zola","cobalt","cobalt.rs","rust","SSG","static site","generator","migration","conversion","switch","gutenberg"],"content_html":"I wish I could have the features of both tools, but for now I will use zola over cobalt. A tiny migration and feature comparison story.
\n\nNote: "gutenberg" got renamed to "zola" in September 2018.
\nWhile cobalt is a pretty nice and easy static site generator written in Rust (in the vein of Jekyll, the famous tool used for GitHub pages), I struggled a bit.\nEspecially that the stylesheet compilation is not enabled by default yet, but also other tiny annoyances or missing features played into that. I still want to stick with Rust and the only other active option here is gutenbergzola. But to be honest, this one also comes with its complications.
Cobalt is your tool for a quick blog-like setup, because it supports the usual two different article types of (static) pages and blog posts. Zola doesn't really have such distinction, and therefore you need to be more elaborate if you want to achieve the same.
\nSince both projects are pretty young, they share similar shortcomings: the template libraries are usually not feature complete yet or behave a slightly bit different than their spiritual role models. Mostly one can work around those issues, but it's a bit painful still.
\nZola shines here a bit brighter, since it supports macros and shortcodes. Especially the latter is something nice, because I can quickly enrich my pages with snippets, which are usually weird or hard to be done purely in Markdown. Embedded HTML is not the nicest thing and if I can abstract them away, I'm all in for that.
\nAnother tiny plus: Zola is supported by Netlify, therefore I do not need to install the binary somehow on their side. Now I can let it build the final site on deployment and skip committing a prebuilt project. Pretty sure I could have done it with cobalt as well, but this way it worked out of the box.
\nSpeaking of tooling: I was a bit adventurous and bundled two tiny binaries anyway and it worked.
\nfd
- the simpler find, written in Rustfd advertises itself as A simple, fast and user-friendly alternative to 'find'.
\nAnd according to their benchmarks it is incredibly fast. But I also like the easier command-line options. The interface is designed for the common cases. And it is written in Rust. Have I mentioned that I love Rust? No? ;-)
\ntidy-html5
- getting rid of awkward whitespaces and indentation\n\nThe granddaddy of HTML tools.
\n
The granddaddy of HTML tools. — well, yes, tidy is a pretty old tool, I probably used it already 10+ years ago.
\nI recommend this helper to anyone who gets slightly shitty HTML generated from their CMS, static site generator or whatevery is producing such files automatically.
\nUsually there is no harm in leaving the HTML pages as they are, as long as they render in most browsers. I like my pages nice and clean. And I wish most programs would come with tidy builtin. I know, I know, I'm such a dreamer.
\nHere a snippet how I use fd
and tidy
for my post processing:
# from my Makefile:\n\nTIDY_SETTINGS = -q -m -w 0 -i \\\n --indent-with-tabs yes \\\n --indent-spaces 2 \\\n --tab-size 2 \\\n --clean yes \\\n --join-styles yes\n\nbuild-tidy-html:\n fd -p public -e html -x sh -c "echo {} && tidy $(TIDY_SETTINGS) {}" \\;\n
\n.html
in the public
folder-x
option; sh -c "…"
tidy
removes all superfluous whitespaces and empty lines, properly indents the tags, and cleans up the inline styles, which is quite handy for the code snippes, because zola (and cobalt) do not use a global/external stylesheet for the syntax highlightingIf you want to learn more about all the possible flags and options, check out the tidy reference for details and explanations.
\nThis step helps me a lot getting a consistent output for my whole site, also I will get warnings/errors in case I messed up something in my templates and markup.
\n\nzola
In the zola repo itself is a nice comparison between zola, cobalt, hugo, and pelican. Since I'm not really interested in non-Rust generators I just keep rephrasing the differences between the first two.
\nThe only thing zola has not is the data files feature of cobalt, which can be seen as flat file databases in YAML, JSON, or TOML format. You would maybe need this if you organize and use a lot of data while site generation step. While it sounds nice it is not something I really need yet.
\nWhat I really appreciate in zola is the wide support of tiny features complementing Markdown based editing. Similar to GitHub wikis you can internally link between documents. Furthermore you get automatically linkable headers and can autogenerate a table of contents (TOC). The template engine is pretty nice (I still need more definable types like arrays and maps, something I could do with liquid based engines). Themes, Aliases, and Pagination are not my main concern yet, but at least the latter could come in handy pretty soon. What I really like are the (custom) shortcodes, I loved it already years ago as a WordPress plugin (or you remember them even from bulletin board/forum software). And the macro system is quite useful when you do base your design on a theme. As mentioned above the stylesheet compilation based on SASS is quite nice, just because splitting up the styles in logical chunks helps to better reason about them.
\nCurrently there is a discussion about custom taxonomies which is interesting.
\nOh and I totally forgot a tiny but really convenient feature: the automatic page reloading while editing and tweaking. People who also build React apps will love it.
\nI have more items I'd like to see:
\nAlthough zola is not cobalt or Jekyll, the move was not as painful as I initially thought it would be.\nThe template languages are similar enough, syntactically and semantically, so that adjustments were sometimes just search-and-replace, both use curlies for their tags.
\nSome of the shortcomings of Tera made some changes slightly more difficult, but overall nothing was totally impossible, or I could just live with a similar but simpler solution for now.
\nI didn't like the builtin rss feed generation and it lacks different feed formats anyway, so I work around this by creation my own feed pages in a slightly awkward way: I built "html" pages for each feed format and rename/move the files in a post processing step later. The sitemap generation was a bit more straight forward.
\nThe biggest part of my time I spent in adopting the blog post directory structure and page traversal for the home page. And while writing this post I have an idea how to make my life easier in the future: since I'm not really using the categories (yet) I could utilize a default category page for simpler blog post listings. Will definitely try this one. (If/when we get custom taxonomies I could shift again to use categories as intended.)
\nThe organization of post related assets is now a bit different, but maybe even easier/more intuitive. Getting the cover image URL became a bit simpler now.
\nOverall the migration was quite fast, less than a day or so. For how it was done I recommend to head over to the markentier.tech repository and check out the commit history.
\n\nRecently Netlify opened up their previous beta features: Forms, Identity, and Functions (Lambdas).
\nI have no immediate use for the forms and identity service (yet), but the functions are probably a nice feature to check out first.
\nAs with most of their tools and support they promise that it will be easy or easier than the underlying solution. And while I haven't even actively played around with AWS Lambda yet, I tested it via Netlify's Functions feature. The free tier is quite limited, of course, but I cannot image a simpler way to get started with functions and lambdas and the whole serverless universe. Although I don't really like this marketing term, since in the end everything is running on some server somewhere. I'd rather think of the evolution after Heroku, which never claimed to be serverless, but a platform where most of the painful infrastructure administration is abstracted away. Functions and Lambdas just further narrow down the scope and surface, but still: they are really tiny applications running on managed servers and services, and a little bit more on demand.
\nAnd I have to admit, there is great appeal in such services. Especially for single person projects (like this site), where you want a little bit more than just pure static, but also do not want to invest too much time and effort in setting things up.
\nBecause in the end all you really want is: getting things done.
\nNote: The tool got renamed to zola
. IIRC there was a naming conflict in the past.
A comment on my previous post »Push, rinse, repeat …« made me think about that you might want to know how this blog is being built and deployed.
\n\nThis site (if you reading this blog post at its canonical location) is freshly hatched. And I realized that I used a couple of things some people might not be familiar with … yet.
\nWhile everything is still in progress, the heavy lifting for getting it up and running is already done, and I think I can maybe give you some insights on how it is made.
\nIn this article we will discover how to get a static site or blog bootstrapped with a GitHub repository, filling it with cobalt.rs, deploying on netlify, using your own domain, get it DNS managed, and using SSL/TLS via Let's Encrypt (https!). A final note on status monitoring via updown.io included as well.
\nSounds exciting? Then keep reading. Beware the long read, it is probably not your typical how-to guide, it is more like a journey.
\nStep Zero would be to have a reason why you want to build a site or blog, but I assume you found your reason already and just cannot wait to get your ideas published into the world.
\nSo especially tech-savvy people love to build or run their own stuff. The middle of a hosted blog and running stuff on your own server is probably using some managed services without relying on the software of the hosting providers. Tools like WordPress are cool and I remember that they are quite beginner friendly, also to the lesser tech-savvy ones, but—no matter your actual background and knowledge—you might have come to the conclusion that you really want to explore the field of owning more of the build and run aspect. You might not need a fully fledged software, maybe even require less to no dynamic features (makes it mostly safer against hacks), or you're just not satisfied with what they offer. But writing your own CMS or blog engine from scratch is also something many developers have tried, but in the end stepped away or replaced this idea with better projects. I always wanted to build such software, but lost interest or motivation in continuing it. A static site usually serves my purposes quite well.
\nAnyway I'm lost in my own thoughts, but you want to read a guide, not my history and reasoning. At least I could save this for a future post.
\nSo, the very first thing is: how and where do I want to store my data (content and site stuff) and which tool do I want to use to build and manage?
\nNowadays a GitHub repository seems to be the most favourite choice. You could use alternative services and even another version control system entirely. Git is quite popular and you will find a lot of help out there when stuck.
\n\nSince we want to build a static site, we need a tool helping us. Of course, you could write and maintain everything manually, but you will get exhausted by it.
\nSo naturally a vast amount of static site generators exists, because a lot of people feel the same. And plenty of them created applications to help us.
\nI went for cobalt, because I want to learn and use Rust more often. If you not so much into such adventures, then the following tools will do it as well: Jekyll, Hugo, OctoPress, Gatsby, Middleman, … (check out https://www.staticgen.com/ to get a more complete list of available static site generators). Jekyll is special, because it is also supported by GitHub Pages, which would be another alternative to deploy your static site.
\n\nWithin your freshly cloned and probably still very empty repository you run something like cobalt init
(or whatever your tool requires) and start the heavy work of creating a website from bits and pieces. You can spend already hours and days just building the site, since layout and design are a neverending story.\nBut also pages and posts need to be written, and this will take some time as well.
</head><body>
Of course, you can start easy. It doesn't have to be pixel perfect right from the start. Look at this one, it might seem a bit rough and edgy here and there (depending when you will discover this article).
\nAnyway, let's assume you learned some lessons around web design and writing. So don't forget to commit and push your changes to your GitHub repository. Do it more often than not. Your computer might store everything now, but it's not reliable in the long run and you don't want to deploy your machine anyway. Building a home server is a completely different story.
\nSo, you built, wrote, pushed. Now we come to the beauty of 2018: let it automatically publish your site and expose it to the world!
\nI discovered netlify just because we use and support it at my work (Contentful), and by googling for this article I just found some resources if you're interested in it:
\nSince I'm not starting with a Contentful-backed approach I will not dive deeper into it. I leave the advertising to our evangelists for now. :-)
\nInitially I really thought that GitHub Pages would be okay already, but netlify comes with some batteries I didn't want to miss, the biggest one was the Let's Encrypt based SSL/TLS certificate, so I can get https
all the time and not have to think about the renewal in the future. This works best if you not only bring your own domain, but also can configure it this way that you can let netlify manage your DNS settings (I think this was recommended for better subdomain stuff and enforcing https across them; could be done in your DNS provider probably, but hey, let's give it a try and see how netlify can deal with it).
Another cool feature is that you can use GitHub's pull requests (and branching in general) to get a preview deployment. So you can check your changes not only on your local machine, but also let other see how the things will look like (pretty good for teams and collaboration). In fact, I used this for this blog post, so I can test this opportunity. Another tiny benefit is that you can use online validators (like https://validator.w3.org/nu/ for HTML) and fix issues before they go into your production site. Sounds cool, he?
\nHey, netlify folks, when you reading this by accident, you can donate a bit to my PayPal account for my advertisement efforts.
\nSetting up netlify is quite easy even without the domain stuff. Point it to your repository and maybe define what (supported) tool should be run.\nSince cobalt
is not widely used yet, I already ship my generated site with the repo, so I only need to tell netlify, which folder to use instead. That would be also the case if you went full manual and craft your pages by hand.
Netlify offers a lot more stuff, which I'm not using yet. You should discover their features on your own. Split testing might be interesting for sites with customers and your attempts to drag them to CTAs (call to actions). Form handling is probably nice if you want to get feedback or collect other data.
\nSorry. You wanted to know how difficult it is to get your site deployed with netlify.
\nWell, I have to disappoint you. Once you've set up your repo, branch and optional configs, you're pretty much done.\nYou can either trigger a deployment manually or just push changes to your previously configured branch.\nYou wait a few seconds (the very first deployment might take a slightly bit longer) and then you check if you're site is up and running.
\nI really mean it, it is so simple, that any How To for netlify is more complicated and longer than their step by step guidance within their interface. Give it a try.
\nIf you're really stuck, then their help pages should do the rest.\nAll the things I wanted to use and try was provided within netlify's documentation. And in the end, deploying a static site should not be so much magic.
\nI know that nothing comes with perfect 100 percent uptime. That's usually okay. And better you're notified about it. Independently, of course.\nWhile you can go with well-known services like pingdom, I use updown.io, which is not free, but with initial signup you will get enough credits to monitor a single website for a long time. If you think it is really not worth it, change to another uptime monitoring solution.
\nWhat I like about updown is, that they offer a simple status page, you can expose via a subdomain you control. For this blog you can check it under https://status.markentier.tech/:
\n\nThat's all I have for today.
\nI hope this article is useful to anyone of you. If you want to give feedback, have questions or suggestions, you can usually find me on Twitter: https://twitter.com/asaaki — See you soon! 👋🏽
"},{"id":"https://markentier.tech/posts/2018/03/push-rinse-repeat/","url":"https://markentier.tech/posts/2018/03/push-rinse-repeat/","date_published":"2018-03-11T18:00:00Z","banner_image":"https://markentier.tech/posts/2018/03/push-rinse-repeat/cover.png","summary":"I haven't bootstrapped a blog or static site for quite some time.","tags":["new","blog","bootstrap","fresh","static site"],"content_html":"I haven't bootstrapped a blog or static site for quite some time.
\n\nWith a service like netlify this becomes kind of a bearable task.
\nAnd since this site hasn't seen much popularity yet—it just exists for less than 24 hours—I can even play this game live and in production.\nLike the good old PHP-FTP times.
\nFor fun I added already some meta tags for social media consumption like Twitter.
\nIt should look like the article image.
\nThis is just the second pretty useless post, but I think this is not so unusal for new blogs.\nBear with me, actual content will follow. Hence this new site. ;-)
\nCrossposted to: dev.to, Medium
"},{"id":"https://markentier.tech/posts/2018/03/hello-code/","url":"https://markentier.tech/posts/2018/03/hello-code/","date_published":"2018-03-10T23:42:00Z","summary":"Nothing but initialization of yet another tech blog.","tags":["hello","world","new","blog"],"content_html":"Nothing but initialization of yet another tech blog.
\n\nThis blog post has no deeper meaning.\nIt exists for sole entertainment of systems, servers, build tools, and the author of this site.
\nclass MarkentierTech < Blog\n def render\n # magic here\n :unicorns!\n end\nend\n\n#=> 567890-abcdefghi-234567890-abcdefghi-234567890-abcdefghi-234567890-abcdef.80\n
"}]}