Swarm: Distributed Computation in the Cloud

Ian Clarke, former lead developer of Freenet, is working on a cool project, named Swarm. The key question that inspired Swarm is: how to distribute data and computation across multiple computers such that the programmer need not think about it?

Based on Scala 2.8, Swarm draws upon the programming language feature “portable & delimited continuations“, i.e. the capability of migrating a piece of a thread to a different computer (“move the computation, not the data“). The promise: let the programming framework handle the distribution problem, instead of having the programmer care about it (aka MapReduce, Databases, …).

GSA Cloud

Funny thing I came across recently: Web-shop for GSA cloud computing apps & services. They have a shopping cart! :)

According to Wikipedia:

The General Services Administration (GSA) is an independent agency of the United States government, established in 1949 to help manage and support the basic functioning of federal agencies. The GSA supplies products and communications for U.S. government offices, provides transportation and office space to federal employees, and develops government-wide cost-minimizing policies, among other management tasks. Its stated mission is to “help federal agencies better serve the public by offering, at best value, superior workplaces, expert solutions, acquisition services and management policies.”

On the Website Apps.gov the GSA pitches their cloud computing offerings:
* Business Apps
* Cloud IT Services
* Productivity Apps
* Social Media Services

Virtual Private Cloud (VPC)

AWS Virtual Private Cloud

Amazon Web Services today announced a new feature (in limited beta): Virtual Private Cloud (VPC).

You can create a VPC that allows secure EC2 integration into existing IP network infrastructure. Assign private IPv4 addresses to the VPC in an address space between /28 and /18 (16 to ~16,000 IPs). The VPC can be partitioned into up to 20 subnets.

The term “private” of VPC refers to the the usage of IPsec. Integration is done with a gateway at the edge of the VPC that can be connected to an existing VPN.

Future

The setup enables end-to-end network security and isolation through IPsec and a user-assigned private IP address range. Right now the VPC is completely isolated. However, plans are to enable an option to open the VPC to the Internet. Next, it will be possible to assign not only an address range but individual IP addresses to EC2 instances. The Amazon folks are also working on traffic filters for the subnets.

Further Reading

The idea has been discussed before by Reuven Cohen and in the paper “The Case for Enterprise-Ready Virtual Private Clouds” (PDF).

New Mechanism to Clean your Mobile Device

Recently, my friend Abraham and I developed a prototype and filed a utility patent for a cleaning mechanism for mobile devices. The mechanism is integrated into the box that is used to carry the device, such as a belt pouch.

Prototype iPhone bag

Update: New prototype (9/29/2009)

Both protect and clean your mobile device

We came up with the idea when we noticed that touchscreen devices, such as digital cameras, iPhone, etc. become dirty quickly due to grease spots and finger prints on the touchscreen surface. Currently available bags and boxes only have limited support for cleaning mechanisms, e.g. a bag with microfiber lining. Such bags suffer from lack of stability and robustness of the outer shell and do not protect the device from physical damage. Therefore, our cleaning mechanism is targeted at boxes, such as belt pouches, which can be equipped with a hard shell that protects the device.

Scetch of the cleaning mechanism

If you are interested in learning more about our patented mechanism or would like to get in touch to purchase a prototypical belt pouch, just drop me a line.

Publication number: DE202009002049 (U1)

Amazon Export

Amazon Import/Export

Amazon Web Services physical data transfer (Amazon Import/Export) allows cloud users to ship storage volumes to Amazon data centers and upload the data into S3. Now, Amazon Export allows users to get their data back via physical data transfer.

Off-Site Backup & Disaster Recovery

The service resembles traditional Disaster Recovery (DR) offerings where a customer sends storage volumes (disk or tape) with backup data to an off-site data center managed by a DR service provider, such as EMC, SunGard, etc.

Fighting Bandwidth Limitations

Amazon Import/Export is especially useful when faced with large amounts of data that need to be moved from the customer site into the (Amazon) cloud. Another obstacle that affects the integration of customer site infrastructure with remote infrastructure “in the cloud” is limited bandwidth.

Have a look at the AWS Import/Export Calculator!

Relaxed Consistency

Since Werner Vogels’ famous blog post & paper about eventual consistency (aka relaxed consistency) some time has passed. Relaxed consistency is a concept related to the complexity of distributed systems. Since it can take some time for a read or write operation to be properly validated and passed through the various layers of a distributed system, the question arises, “how does this affect user experience?” Clearly, not in a good way.

Werner Vogels explains that for certain applications it might be necessary to tolerate a certain amount of system inconsistency in order to speed up the user-perceived time it takes to complete an operation (such as adding an item to the famous shopping cart). The concept is somewhat similar to Ajax, however, Ajax happens on the presentation layer whereas relaxed consistency happens somewhere in a back-end layer of the software stack.

Another interesting paper, Yahoo! PNUTS, proposes something in-between strong consistency and relaxed consistency, pointing out that we are not taking about a binary choice (be consistent or not) but rather about a continuum of possibilities. Again, it depends on the application characteristics. The Yahoo! researchers discuss an amusing scenario:

First operation: Remove mother from access list to shared picture album.

Second operation: Upload spring break pictures.

The authors argue that it would lead two a rather unpleasant user experience if these two operations could take place in reverse order (as it might be possible in Amazon’s relaxed consistency model). They propose a “per-record timeline consistency: all replicas of a given record apply all updates to the record in the same order.”

For more information have a look into the excellent papers in the refrences.

References:

http://www.allthingsdistributed.com/2007/12/eventually_consistent.html

http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

http://research.yahoo.com/files/pnuts.pdf

Opera Unite

Opera 10 introduces a technology called Opera Unite, that extends your browser to become a Web server. The idea is to enable non-technical people to serve and share content and services from their local computer, such as file sharing, chat rooms, Web hosting, remote access to music files, etc.

In a way, Opera Unite is an opposite trend to Cloud Computing where content is distributed and services are provided from large-scale data centers under the administrative domain of single providers, such as Amazon or Google. Instead, Opera Unite is a technology that borrows from ideas known in P2P and Grid Computing where hybrid user/provider models are common.

I have been a big Opera fan ever since due to the browser’s incredible speed (much faster and less resource-hungry than Firefox, at least on my system). Opera Unite could be the game changer that Opera needs to compete with IE and Firefox. However, it is only a matter of time until Google integrates GTalk, Picasa, etc. with Chrome and Microsoft integrates Office Live with IE in a way that is similarly user-friendly. Back to the Clouds?

Opera has taken a lead in browser-based file sharing that could pay off very soon. In previous versions, Opera already offered a neat built-in Bittorrent client. Now, with Opera Unite, there are much more file-sharing features: easy access control (public, groups, private), easy file system integration, etc. Great job!

Other services make less sense to me, like Web hosting. Only few people are always online (if so, I don’t want to see the electricity bill…) and therefore cannot provide even moderate availability.

Zotero plus JungleDisk helps you manage your research sources

Today I tried out Zotero 2.0 beta in combination with JungleDisk@S3. Zotero comes as a Firefox plug-in that allows you to gather and organize information sources, such as research papers and blog posts. You can also sync local files, e.g. PDF documents, to a WebDAV server. First I looked into free solutions, such as GMX Media Center or setting up a local WebDAV FTP server. However, I came to the conclusion that using JungleDisk and Amazon S3 would be much easier and pricing is also not too expensive ($2/month for a JungleDisk subscription plus Amazon’s prices for S3). Set-up cost me 10 minutes on my private Ubuntu laptop and another 10 minutes on my Vista work Desktop. Now I have everything accessible from anywhere + highly reliable back-up in Amazon’s infrastructure.

Will the Cloud become a Common Good?

The Common Good

In economics the term “common good” refers to a good that is rivalrous but not excludable. Fish is an example for a common good that (assuming there is no fishing regulation) is not excludable because anyone with a boat can go fishing. However, it is rivalrous since there is a limited number of fish in the oceans – and in fact overfishing is a problem in many regions of the world.

The Internet as a Common Good

I stumpled upon a EU document claiming the Internet was as a common good and therefore it would be necessary to reform Internet governance (PDF). The author argues that the market alone was not capable of controlling a global public-private infrastructure of communication, that the Internet is largely based on non-proprietary and open technologies and that the preservation of freedom rights would demand a reform of current Internet governance.

Although I do not agree with the author and au contraire believe that self-regulation and market mechanisms (together with standards bodies, such as W3C) have worked quite well in the past and will work in the future (as long as net neutrality is preserved), the discussion about private common goods / global public goods has drawn my attention.

In fact, the Internet seems to fit the common good attributes, as it is rivalrous (net neutrality discussion), yet not-excludable.

The Cloud as a Common Good

Now, thinking about Cloud infrastructure in lieu of the Internet, i.e. Amazon Web Services, Google AppEngine, Microsoft Azure, etc. Currently these services does not fit the non-excludability propery and therefore would rather be categorized as private goods. However, initiatives such as the OpenCirrus Testbed point to a different direction. If the Internet is considered a common good, the Cloud might become a common good as well.

In fact, I believe the Internet (primarily being a communication infrastructure) and Cloud services might blur and melt together into a combined communication and middleware infrastructure that will be much more powerful than what we have seen so far. Just think what kinds of applications you could build if you could rely on processing power, storage, messaging and so on like you do with basic communication (TCP/IP, HTTP, etc.). People who think this is just hype, well, they will miss a big opportunity :p

From Push to Pull

I recently came across a fascinating working paper by John Hagel: From Push to Pull (2005). Hagel’s work focuses on emerging approaches for mobilizing resources. He states that we are living in a “push” world, whereas new business opportunities and technology innovation arise from “pull” models.

Push

Push resources to areas of highest anticipated need, e.g.

• in education: standardized curricula
• in business: highly automated plants and service platforms to deliver resources to the right place at pre-determined times
• in technology: massive enterprise applications to deliver IT resources and services at pre-determined times
• in media: traditional broadcast TV

Pull

On the contrary, pulling resources allows people to mobilize resources when they need them, e.g.

• in education: e-learning, virtual classrooms
• in the production industry: Toyota (lean production, JIT, etc.)
• in technology: community-driven OSS development
• in media: Blogs, collaborative media creation, VoD, etc.

Cloud Computing, i.e. on-demand, self-service IT technology (IaaS, PaaS, SaaS, etc.), is the newest member in this concert of promising “pull” approaches. Similar to the other “pull” examples, Cloud Computing is a disruptive technology shift, driven by growing uncertainty, intense competition and the inability of “push” models to fulfill customized wishes in a timely manner.