Blueprint 3.4 is out with two powerful additions: templating and rules.

Template functionality was included in release 3.3 and makes Blueprint a more complete configuration management system. Simple template use enables the substitution of system parameters such as the number of CPU cores, memory, domain names and IP addresses, in config files. More complex cases include iteration over output from a shell command.

Here's a quick example where we configure Unicorn to use four workers per CPU in /etc/unicorn.conf.rb.blueprint-template.mustache:

worker_processes {{`expr 4 \* $CORES`}}

Rules were part of 3.4 and are analagous to the blueprintignore concept. Power users will eventually want to pay specific attention to, instead of ignore, a set of resources. blueprint-rules solves that problem.

Check out what we include example.blueprint-rules:

:source:/usr/local
 /etc/init/example.conf
 /etc/nginx/sites-*/example
 :package:apt/libmysqlclient-dev
 :package:apt/mysql-client-5.1
 :package:apt/nginx-common
 :package:apt/nginx-light
 :package:apt/ruby-dev
 :package:apt/rubygems
 :package:rubygems/*
 :service:sysvinit/nginx
 :service:upstart/example

Now create the blueprint example using rules and it will only include those resources:

blueprint rules example.blueprint-rules

Lastly, we've revamped Blueprint's documentation and think you'll find it easier to get answers fast. We hope you enjoy this release and the progress we're making on the project. As always, we're here for any questions or suggestions. Enjoy!

Permalink | Leave a comment  »

Blueprint 3.2.0 is out and comes with two new commands and two new arguments.

First, hot on the heels of our announcement that blueprints work directly with AWS CloudFormation, we've added --cfn to blueprint-create and blueprint-show to generate a complete CloudFormation template from a blueprint.  The template contains all the necessary pieces to create a single EC2 instance running Amazon Linux that'll apply your blueprint at provision time.  From there you can of course add more resources to build out a larger infrastructure with CloudFormation.

Next is the --diff=subtrahend argument to blueprint-create, making it easier to create compact blueprints.  Observe:

blueprint create --diff=base custom

is roughly equivalent to

blueprint create dummy
blueprint diff dummy base custom
blueprint destroy dummy

but doesn't actually save dummy to disk.  The practice of creating base system blueprints and compact customizations on top of that is gaining traction and now it's a whole lot easier.

Last are two commands, blueprint-split and blueprint-prune, that make creating modular blueprints (like a base blueprint and a customization) possible without crazy blueprintignore(5) incantations.  blueprint-split interactively creates two blueprints from the resources contained in one source blueprint.  blueprint-prune works much the same way but creates one blueprint from a subset of the resources in a source blueprint.

As always, the full details are on GitHub: https://github.com/devstructure/blueprint/compare/v3.1.0...v3.2.0

If you're not yet using Blueprint, the installation instructions in the README will get you started.

Permalink | Leave a comment  »

Just last week, Amazon Web Services rolled out the first production release of the Amazon Linux AMI and with it some powerful new features for CloudFormation.  There’s a lot there (like updating stacks and IAM integration) but we at DevStructure are most excited about Application Bootstrapping.

The new Amazon Linux AMI comes with the aws-cfn-bootstrap package, which can unpack tarballs, place configuration files, install packages, and restart services at provision time.  Sound familiar?  That’s right, this new section of the CloudFormation template language uses the blueprint(5) file format!

Amazon’s PDF whitepaper, Bootstrapping Applications via AWS CloudFormation, walks through building a complete stack template by hand.  To summarize, you’ll need to create an IAM resource and declare your EC2 instances like this:

"Resources": {
  "DemoInstance": {
    "Metadata": {
      "AWS::CloudFormation::Init": {
        "config": THIS IS WHERE THE BLUEPRINT GOES!
      }
    },
    "Properties": {
            "ImageId": {"Fn::FindInMap": [
              "AWSRegionArch2AMI",
              {"Ref": "AWS::Region"},
              {"Fn::FindInMap": [
                "AWSInstanceType2Arch",
                {"Ref": "InstanceType"},
                "Arch"
              ]}
            ]},
      "InstanceType": {"Ref": "InstanceType"},
      "KeyName": {"Ref": "KeyName"},
      "SecurityGroups": [{"Ref": "DemoSecurityGroup"}],
      "UserData" : {"Fn::Base64" : {"Fn::Join" : ["", [
        "#!/bin/sh\n",
        "/opt/aws/bin/cfn-init",
        " -s '", {"Ref" : "AWS::StackName"}, "'",
        " -r 'DemoInstance'",
        " --region '", { "Ref" : "AWS::Region" }, "'",
        " --access-key '", {"Ref": "DemoKey"}, "'",
        " --secret-key '", {"Fn::GetAtt": ["DemoKey", "SecretAccessKey"]}, "'",
        "\n",
        "/opt/aws/bin/cfn-signal",
        " -e $?",
        " '", {"Ref" : "DemoWaitConditionHandle"}, "'",
        "\n"
      ]]}}
    },
    "Type": "AWS::EC2::Instance"
  }
}

The user-data calls cfn-init with the newly-generated IAM credentials to fetch and process the metadata, and cfn-signal to report success or failure via a WaitCondition.

Packages managed by Yum, Python’s easy_install, and RubyGems plus files and services all work natively within CloudFormation.  Source tarballs will work if you upload them someplace and provide the fully-qualified URL.

Going the other direction, the metadata from an existing CloudFormation template can be loaded into Blueprint by copying out the "config" document fragment and passing it on standard input to blueprint-create(1).

Our thanks to the Reto Kramer, Chris Whitaker, and Adam Thomas for making it even easier to deploy blueprints to Amazon EC2.

Today, we’re releasing Blueprint 3.1, which includes a number of fixes and improvements but most importantly, understands all of Amazon’s extensions to the blueprint(5) format, allowing seamless transition to and from AWS CloudFormation.  Get 3.1 from GitHub, DevStructure’s Debian archive, or from PyPI.

Permalink | Leave a comment  »

Just in time for the Node Knockout, I'm happy to announce preliminary support of Node and npm in the master branch of Blueprint.

For the uninitiated, installing Blueprint is easy:

git clone git://github.com/devstructure/blueprint.git
cd blueprint
make && sudo make install

Creating a blueprint of a server and generating shell code that can be used to configure the production server the same way is likewise easy:

blueprint create -S knockout
# Run knockout.sh in production.

If you're new, check out the Blueprint tutorial while you're getting started.

The details are on the blueprint-users mailing list post but the short version is to use Chris Lea's PPA (node.js or node.js-devel) to install Node itself and everything else will work out just fine.

Happy hacking!

Permalink | Leave a comment  »

Those of you that are fans of RedHat, CentOS, or Fedora are in for a
treat today: 3.0.8 is all about you.

This release closed only three bugs (one of which it caused along the
way!) but made 1000 additions and 173 deletions and made first run on
RPM-based systems on the order of 1000 times faster.

Compare: https://github.com/devstructure/blueprint/compare/v3.0.7...v3.0.8
Tree: https://github.com/devstructure/blueprint/tree/v3.0.8

Functionally, nothing much has changed, so keep on keeping on.

Permalink | Leave a comment  »

Big news, the 3.0.7 release of Blueprint now includes the blueprint-push and blueprint-pull commands, previously part of Blueprint I/O.  We decided to roll these commands into the main release for simplicity and exposure. Additionally, this release closed 13 issues and contains 42 commits.  

Highlights include:

• Commands for looking inside blueprints, with an eye towards scriptability: blueprint-show-sources(1), blueprint-show-files(1), blueprint-show-packages(1), blueprint-show-services(1), and blueprint-show-ignore(1).  These make it far easier to understand what's being included in blueprints without having to squint at raw JSON.
• A command for interacting directly with blueprints stored locally in Git: blueprint-git(1).
• Blueprint is now less aggressive about using sudo in blueprint-apply(1) in order to play nicely with non-interactive invocations.  Keep using sudo yourself if that's typically how you gain root access.

Updated packages are on packages.devstructure.com and PyPI.  Let us know if you have any questions. Enjoy!

Permalink | Leave a comment  »

Cross-posted from the blueprint-users and blueprint-dev mailing lists.

We’re back with Blueprint 3.0.6, featuring 87 commits to master and 13 issues closed since 3.0.5.  Many of those commits are behind-the-scenes, refactoring to lay the groundwork for a more maintainable future.  That doesn’t mean there aren’t some cool features, though.

First up is the much improved support for the various ways people use sudo to gain temporary root access and root shells through -i.  I expect most if not all strange GitError exceptions talking about lack of permissions to be a thing of the past.

Next, ever-smarter defaults for ignoring files to keep your blueprints free of noise: Blueprint won’t bother you about things you didn’t change in /etc/pam.d, about Debian's CD-ROM, or backup files.

blueprint-create, blueprint-show, and blueprint-apply all now support the --relaxed option that generates code without strict version numbers so any version of a package will suffice.  This is particularly helpful (and recommended) for users that don’t mirror their distro’s package archive.

And finally: service resources.  Now Blueprint will track down services that are running plus the files and packages that influence them.  These translate directly to service resources in the generated Puppet and Chef code.  In the generated shell code, changes to any file or package will cause the corresponding services to be restarted at the end.  Now you can use blueprints all by themselves to deploy your environment and application to production.

As always, Matt and I are listening for issues and feature requests, and would love to know how you’re using Blueprint.

• Source: https://github.com/devstructure/blueprint
• Changes: https://github.com/devstructure/blueprint/compare/v3.0.5...v3.0.6

Permalink | Leave a comment  »

I'm in the thick of moving and it's stressful. Two Men and a Truck makes it easier. True to their name, a couple guys show up, pack all of your stuff in a truck and move it to the new location. Building a server is stressful too. What packages do you need? How should you tweak configuration files? Is everything secure? Is the process documented? The list goes on. We want to make it less stressful for you and demonstrate the power of Blueprint. 

Introducing Two Men and a Server - tell us what you want, be as specific as possible, and we'll build a server for you. We'll package everything up in a Blueprint that you can easily reuse on a new or existing server. Ubuntu, Debian, Fedora or CentOS - your choice. AWS, Rackspace, Linode, Slicehost - anywhere you have root. Rethinking application design, moving providers or starting from scratch - not a problem. Just tell us what you need and we'll build it with Blueprint. Richard and I know servers and are happy to help - no strings attached. Get in touch and we'll get you a server.

Permalink | Leave a comment  »

Last week we rolled out Blueprint 3.0.4 to bring the speed and a rich new syntax for ignoring files and packages when creating blueprints. Not to rest on our laurels, we released Blueprint 3.0.5 this week which added the ability to tar up arbitrary directories along with your blueprint and a new blueprint-diff(1) command.

Now we're excited to announce something a little different: Blueprint I/O, a way to move your blueprints around.

Blueprint I/O comes with a default Blueprint I/O Server that we're making available free of charge which stores your blueprints in Amazon S3. Use blueprint-push(1) and blueprint-pull(1) to centralize your configuration management and bootstrap new servers painlessly.

Blueprint I/O is open-source and we're very interested in seeing where you take it.

Permalink | Leave a comment  »

Cross-posting once again from blueprint-users and blueprint-dev (have you joined the list yet?), I'm happy to announce Blueprint 3.0.3, our first release that includes support for RPM-based distros.

Packages managed by Yum/RPM are now first-class citizens just like Debian packages, Ruby gems, Python eggs, and PEAR/PECL packages. In addition, when Blueprint searches /etc for configuration files, it will check the RPM database and ignore files that haven't changed from their packaged version.

What are you waiting for? Install Blueprint from source, our Debian archive, or PyPI now!

Permalink | Leave a comment  »

I just sent a much-anticipated email to the blueprint-users and blueprint-dev mailing lists announcing Yum/RPM support.

Preliminary support for Yum/RPM-based Linux distributions is in the master branch on GitHub. Here's what changed:

https://github.com/devstructure/blueprint/compare/8b6db6f5dceffbae9bd0ffb4646723c1c49aea15...master

The two outstanding issues in the Yum/RPM world now are:

* Extra items in the default ignored-files list.
* Python 2.4 support for CentOS.

Thanks especially to Kal McFate for the initial work and to everyone who helped out with RPM commands and output along the way.

I've closed the original issue #5 so please open issues on GitHub for any problems you experience.

If you've been waiting for blueprint to support your favorite RPM-based distro, now's time time to see how easy it can be to reverse engineer your servers.

Permalink | Leave a comment  »

We're super excited to see that blueprint was chosen as the featured project in the new Github Reflog. It's like an Oscar for open-source geeks. Kind of? Maybe just a little bit? Ok, nevermind. Anway, a big thanks to the Github guys and we're looking forward to the weekly reflog!

This is one of the coolest projects I've seen in a long time. 

Simply put, Blueprint reverse engineers servers. It scans your system, records installed packages from various managers (aptitude, ruby's gem, python's pip, etc), archives software built from source, and records configuration file changes. It then packages your entire server configuration into a bootstrap.sh file with an attached archive. You can take that bootstrap package and run it anywhere to replicate your server setup.

Permalink | Leave a comment  »

We've gotten emails from potential contributors about the recently open-sourced blueprint tool. In an effort to centralize that discussion, we've setup 2 Google Groups. If you're interested in contributing, have ideas about blueprint or just need help, this is a great place to start. See you there!

Blueprint Users - http://groups.google.com/group/blueprint-users

Blueprint Dev - http://groups.google.com/group/blueprint-dev 

Permalink | Leave a comment  »

It's been a while since we've talked, but we haven't forgotten about you. However, things are changing and we want to bring you up to speed.

What's up?

Richard and I are obsessed with simplifying DevStructure. Our service has always revolved around 2 components - Blueprint and Sandbox. We've revamped Blueprint and Sandbox as standalone projects. And we're open-sourcing them, Blueprint today and Sandbox to follow.

Why are you doing this?

Two reasons. First, we want to make it as easy as possible to use our tools. Second, we want people to see, understand and contribute to how the software works. DevStructure accounts are no longer required and everything operates via the command line.

The new Blueprint

Blueprint reverse engineers a server. You work, like you always have, via the command line. Install system packages, language packages, source or modify configuration files. We'll work backwards to figure out what you did and store this information in a "blueprint". With a blueprint, you can recreate the environment on another server or export the configuration to Puppet or Chef.

We plan to continue improving the brains behind Blueprint to more accurately detect system changes. Currently it works only with Ubuntu servers, but we've started thinking about Redhat based systems.

Show me the code

Now you're in the loop. As always, if you have any questions please let us know. Otherwise check out the code and get started today.

• Source code
• Issue tracker
• Documentation

Permalink | Leave a comment  »

It's been a busy month at the (virtual) DevStructure office. We've listened to feedback, learned from how you're using our tools and thought about the direction we should be heading. Today we're announcing several changes, a reboot of sorts, designed to streamline your workflow and decentralize our service. 

1. Blueprints are now stored in local Git repositories, with an option to push them to DevStructure. We think of blueprints as changing snapshots of your server, so Git is the perfect backend.
2. A simple, local web interface for our sandbox and blueprint tools. A small Django app runs on each server, delivering an overview of local sandboxes and blueprints and an alternative to the command line interface.
3. We no longer actively manage your servers via Puppet or login remotely. Instead, updates run via a cron job that polls our servers for new releases.
4. It's free. We're no longer counting servers or monitoring billing plans, install it on all your servers!

Those are the biggies. Take a look and let us know what you think! Stay tuned for open source news, screencasts and more software updates. Enjoy!

Permalink | Leave a comment  »

Richard and I understand that with DevStructure, we're asking customers to make a few leaps. There are new tools to learn in blueprint and sandbox, a webapp for bootstrapping servers and the concept of working in a remote dev environment. We're doing our best to minimize these obstacles and blend into your workflow, but working remotely is a frequent deal-breaker. If you want to code in a replica of your production environment and aren't comfortable with vi, your options are limited. Doubledown, our open-source project for fast file syncing, Dropbox or even SFTP can alleviate some pain, but for many users it's simply too big of a disruption to overcome. And if you're using a service provider, there is the additional cost of a dev server.

Virtualization delivers an obvious solution to this problem and a project that caught our eye was Vagrant. It's an open-source tool for building and running local development environments using VirtualBox. The spirit of Vagrant mirrors many of our beliefs, namely that as developers we should build, write code in and export reusable environments. Vagrant provides a command line wrapper for VirtualBox, enabling you to run a Linux server on your local machine. Your code and the infrastructure behind it run Linux, but you can use your favorite editor to hack in a shared folder. It's the best of both worlds: local speed, local editors, a real development environment, zero costs and no duplication since you can reuse all of your work in production.

Today we're pleased to announce DevStructure Boxes for Vagrant. That means you can build, code in and configure a local Linux server, then use our tools to export your work as a blueprint. Once blueprinted, it's ready for deployment using Puppet or Chef. Vagrant servers are free to use on your DevStructure account. We're anxious to get your feedback and hope this integrates into your coding rituals. Let us know what you think!

Vagrant for DevStructure

devstructure32.box

devstructure64.box

Permalink | Leave a comment  »

Doubledown 0.0.2 is here with several new features and fixes thanks to some awesome users.  The build system's better thanks to Adam Vandenberg.  Kellan Elliot-McCrea reported on some undocumented (poor) assumptions and quirks in OS X 10.6 Snow Leopard's ssh-agent(1).  John Dewey submitted a first crack at a Linux version which is in, albeit in slightly different form.  Mike Malone spent some time debugging a mysterious crash with me and the result is support for specifying a non-standard private key.

Thanks to these fine folks for their efforts.

Doubledown 0.0.2

Permalink | Leave a comment  »

In case you missed the tweet last week, we added Chef support to Blueprints. If you're just getting started with infrastructure automation tools, it can be daunting. Blueprints are an easy way to jumpstart your efforts. A blueprint figures out what you did to a server and converts that information into a Puppet or Chef config file or a POSIX Shell script. We had a great conversation with Adam Jacob, CTO of Opscode, and hope to add more Chef integration soon. Ping us with any questions!

On another note, we've updated our home page based on your feedback. We hope it more clearly explains the DevStructure workflow. We understand that our customers are investing time in learning our tools and are extremely grateful that you're helping us make it easier for others to join the club. Thank you!

Permalink | Leave a comment  »

This afternoon we pushed a few final commits, nixed the invite codes, updated the documentation, looked left, looked right and cautiously emerged from our beta shell. Virtual high-fives for all and a huge thank you to our early testers for their invaluable feedback. If you didn't get a chance to kick the tires, no worries - the first month is free and you can sign-up here.

Permalink | Leave a comment  »

DevStructure is about making the development process better, not different. Everyone has their favorite editor or IDE and a comfortable workflow centered around it that makes them happiest. We want to make DevStructure fit naturally into every workflow, bringing configuration management along for the ride.

To that end, we've evaluated many of the market's solutions for remotely editing source code and found ourselves wanting something more.

The class of software we'll call remote editors generally suffers from performance problems. Because all filesystem operations incur a network round-trip, operations like searching your entire project or browsing the entire directory tree are too slow to be used frequently. This category includes editors like BBEdit, Sublime Text, and gedit plus software like Expandrive, Transmit, and SSHFS.

Software like Dropbox performs more favorably because it sychronizes a local copy of your files with the remote copy by way of their service. It's fast and reliable but, like many of the best software above, doesn't always come free.

The choice of workflow and software you use alongside DevStructure will always be yours and today we're adding a new choice in the form of a free, open-source file syncing utility called Doubledown for Mac OS X 10.5 and newer.

Download Doubledown 0.0.1 0.0.2 for Mac OS X 10.5 and newer

Doubledown keeps a complete local copy of the remote directory you're syncing so all your local operations are lightning fast. After it performs an initial sync (being careful not to clobber any local changes), Doubledown is notified of changes by Mac OS X's FSEvents framework and responds by creating, uploading, and removing files or directories as required.

Check out doubledown(1) and the source code on GitHub.

Permalink | Leave a comment  »