Backblaze Developer Newsletter—November 2023

Our monthly newsletter brings technical articles, events, and news to developers working with Backblaze B2 Cloud Object Storage.

Hi there!   This month we lead with some great news: we're now leveraging SSDs in the upload path to accelerate performance for file uploads of 1MB or less. We've seen some great results in our testing, and we're already hearing very positive feedback from customers and partners. Also this month, a deep dive into how to do load balancing in the Backblaze B2 Cloud Storage Platform, another cool new B2 CLI feature, and lots more! We always love to hear about topics that you'd like to see us cover here in the newsletter, on the blog, or in the documentation. Email your suggestions to us at evangelism@backblaze.com (or just reply to this email). How We Achieved Upload Speeds Faster Than AWS S3 In the past, when a customer uploaded a file to Backblaze B2, the data was written directly to multiple hard disk drives (HDDs). Now, for files of 1MB or less, we write the incoming data to those same HDDs and also, simultaneously, to a pool of solid state drives (SSDs) we call a "shard stash," returning a response to the client application once the SSD writes are done. Since writing data to an SSD is much faster than writing to HDDs, the net result is faster uploads. In fact, in our tests, we observed that the Backblaze U.S. East region performed faster than the corresponding AWS region for small files. In this blog post, Backblaze Chief Technical Evangelist Pat Patterson provides a detailed explanation of this milestone performance improvement.   Read All About It → Upgrade Rclone to v1.64.2 or Later to Enjoy a Significant Performance Boost with B2 As we reported in last month's newsletter and linked blog post, rclone v1.64.0, released in mid-September, included a new implementation of multithreaded data transfers, promising much faster data transfer of large files between cloud storage services, including Backblaze B2. However, a bug in that version's Backblaze B2 backend implementation prompted us to advise readers to wait for a fix before upgrading. We're happy to report that our tests verified that rclone v1.64.1, released in mid-October, fixed the issue observed in the previous version. Rclone v1.64.2, released soon afterwards, fixed a couple more minor bugs in the update and build processes. Go ahead and upgrade with confidence!   Find Out More → New Official Docker Image of the B2 Command Line Tool With 3.12.0 of the B2 Command Line tool there is now an official Docker image, encapsulating b2cli and its dependencies. This removes the need to download and install the CLI to local disk, allowing for some very useful automation techniques. For example, building on the compressed, encrypted database backup example we showed in last month's newsletter, we can write the database backup direct to B2 Cloud Storage without writing any intermediate files, or installing any software, to the local disk:   export B2_APPLICATION_KEY=<my application key> export B2_APPLICATION_KEY_ID=<my application key id> pg_dump mydatabase \ | gzip \ | openssl enc -aes-256-cbc -salt -pbkdf2 -pass file:passphrase.txt \ | docker run -i --rm -e B2_APPLICATION_KEY -e B2_APPLICATION_KEY_ID backblazeit/b2:latest upload-file --quiet my-db-backups - backup.sql.gz.enc   Read the Docs → b2get: a Really Simple CLI Utility for Downloading Files from Backblaze B2 Backblaze B2 has an officially supported CLI, plus Python, Java and Go SDKs, as well as compatibility with the AWS CLI and suite of SDKs. That said, it's always interesting to see community-driven projects that fill specific niches. One such is b2get, described by its author, Luke Seelenbinder, as "a really simple CLI utility for downloading files from Backblaze B2." Written in the Rust programming language, b2get is distributed as a single executable file that does precisely one thing: download a single file from Backblaze B2. If it sounds like this tool fits into your workflow, you can find more details at its Lib.rs page or GitHub repo.   Check It Out → Load Balancing 2.0: What's Changed After 7 Years? The Backblaze B2 Cloud Storage Platform now processes two billion transactions a day, over 23,000 transactions every second, with a fleet of load balancers routing incoming requests to over 7,000 production servers. This blog post by Backblaze Senior Site Reliability Engineer (SRE) nathaniel wagner, explains how the evolution of the platform—including the introduction of the S3-Compatible API—has changed our load balancing architecture since Backblaze B2's debut in 2016. There's a lot to learn–I particularly appreciated the explanation of layer 4 load balancing with direct server response (DSR)!   Dive Into the Detail → Know anyone who might like this newsletter? Forward this email to them so they can subscribe! Backblaze, 201 Baldwin Avenue, San Mateo, California 94401 Unsubscribe Manage preferences