{"id":23,"date":"2024-11-26T21:14:25","date_gmt":"2024-11-26T21:14:25","guid":{"rendered":"https:\/\/xxv.owo.mybluehost.me\/home\/"},"modified":"2026-09-13T23:27:32","modified_gmt":"2026-09-13T23:27:32","slug":"home","status":"publish","type":"page","link":"https:\/\/aspen-ddp.org\/staging\/5074\/","title":{"rendered":"Home"},"content":{"rendered":"\n<div id=\"area-of-practice\" class=\"nfd-container nfd-p-lg nfd-theme-white nfd-bg-surface is-style-nfd-theme-white wp-block-group is-layout-constrained wp-block-group-is-layout-constrained\">\n<p class=\"has-text-align-center wp-block-paragraph\"><a href=\"https:\/\/github.com\/aspen-ddp\">https:\/\/github.com\/aspen-ddp<\/a><\/p>\n\n\n\n<div style=\"height:14px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading has-background-color has-text-color has-link-color wp-elements-1\">TL;DR<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"\">Aspen is a general-purpose platform for building various distributed systems<\/li>\n\n\n\n<li class=\"\">Takes a new approach to organizing and managing data at scale<\/li>\n\n\n\n<li class=\"\">Provides distributed system architects with a new set of design tools<\/li>\n\n\n\n<li class=\"\">Provides an adaptable run-time environment that easily adjusts to change<\/li>\n\n\n\n<li class=\"\">Focuses on providing &#8220;good&#8221; performance across a broad range of domains rather than &#8220;great&#8221; performance in just one<\/li>\n\n\n\n<li class=\"\">Provides AmoebaFS, a distributed file system that serves as Aspen&#8217;s first practical use case.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading has-background-color has-text-color has-link-color wp-elements-2\">Motivation<a href=\"https:\/\/github.com\/aspen-ddp\/aspen#motivation\"><\/a><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Aspen solves a problem you probably didn&#8217;t realize we have and that&#8217;s the lack of a general-purpose distributed data platform for building various kinds of distributed systems. To date, if you want to build a distributed system, you essentially have two choices. You either start from scratch or you use composition.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In the first case, you build a custom-purpose distributed system from the ground up that solves all the hard distributed data problems like replication\/erasure-coding, consensus, transactions, version management, drive failure recovery. etc. The advantage is that you get a system ideally suited to your use case and that gives you the best possible performance but the up front cost is enormous.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The alternative is composition where you build your system on top of an aggregation of existing systems. Say CockroachDB for the database, S3 for bulk data, Kafka for event streaming, and Redis for caching. This is the far more popular option as it is usually faster and cheaper than staring from the ground up. However, there are a couple drawbacks to this approach. The primary one being that these systems aren&#8217;t designed to integrate with one another. You can&#8217;t, for example, update an object in S3 and record the update in CockroachDB a single, atomic transaction. So if you need an atomic operation that spans components or otherwise need guarantees in behavior between the boundaries of components, you need to engineer a custom solution for each case and validate that it works across all failure modes of each component. For simple applications this usually isn&#8217;t too much of an issue but as application complexity grows, these work-arounds can become a significant burden. Another challenge with the composition approach is operational support. Your ops team must fully understand the idiosyncrasies of each component, all of which are wildly different from one another, in order to plan, operate, and maintain each system individually as well as your application as a whole.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Aspen aims to provide a third option for distributed systems architects. One that provides many of the benefits of the ground-up approach while simultaneously reducing some of the burden inherent to composition. To do this, Aspen focuses on solving the distributed data management problem in a new, general-purpose manner that emphasizes flexibility in terms of architectural design and runtime operation.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading has-background-color has-text-color has-link-color wp-elements-3\">What it is<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/github.com\/aspen-ddp\/aspen#what-it-is\"><\/a>Aspen is a general-purpose distributed data management platform that is intended to serve as the foundation for building a wide range of distributed applications. It solves the difficult distributed data management problems like transactional updates, recovering from disk failure, and gracefully handling growing and shrinking hardware footprints in a way that provides a high degree of flexibility to both application designers and runtime operators.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Like all systems, the design choices underpinning Aspen make it better suited to some applications than others. Rather than try to optimize for ideal performance in a narrow range of use cases, Aspen instead aims to provide &#8220;good&#8221; performance across a wide range of application domains. The primary goal of Aspen is to provide distributed system architects with a new set of tools with which to address problems.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In addition to opening the door for system designers to go off in new directions, Aspen can also be used to build the kinds of systems we already use today. You can use it to build a traditional database, S3 storage system, distributed file system, message broker, event sourcing application, etc. Due to Aspen being designed for general-purpose use rather than tailored for optimal performance in a specific domain, these kinds of systems built on top of Aspen probably won&#8217;t be quite as performant as their traditional counterparts. The obvious question of &#8220;then why bother?&#8221; arises and the answer is threefold.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>First:<\/strong> Not every deployment needs maximum performance. In fact, the vast majority do not and &#8220;good enough&#8221; is usually just that.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Second:<\/strong> Operational simplicity. All applications built on top of Aspen share the same underlying operational model. There would be little difference between deploying and maintaining a distributed file system built on Aspen than there would be an event sourcing system or database.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Third:<\/strong> (The main one) Systems built on top of Aspen are much more easily integrated as they share the same operational, data, and transaction models. If you need an all-or-nothing atomic operation that updates a database entry, deletes an file in a distributed file system, and adds an event to a stream&#8230; you can do that. And you can do it naturally, no complex shenanigans required.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In short, Aspen aims to fill a role in the distributed data world similar to the role scripting languages serve in software development. Scripts can be used to quickly and effectively tackle many problems that would be difficult to solve with a lower level language. There&#8217;s a reason there aren&#8217;t any C\/C++ web content platforms competing with Django. Python is just better suited to solving that problem and its performance is more than adequate. Aspen is aiming at a similar niche with distributed data arena.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading has-background-color has-text-color has-link-color wp-elements-4\">How it&#8217;s different<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/github.com\/aspen-ddp\/aspen#how-its-different\"><\/a>Of course, Aspen isn&#8217;t a silver bullet. At the end of the day, it&#8217;s just another tool in the toolbox. So, the question is how is it different and what is it good for? To answer that, we must first describe the basics of how it works.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">At it&#8217;s core, Aspen is an object storage system that runs on top of the JVM. All operations are carried out through transactions that can update multiple objects simultaneously with ACD semantics from the standard ACID model (Isolation is achievable but deferred for the moment).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Object data is stored in logical DataStores that support pluggable backend implementations, such as RocksDB and flat files. DataStores are not fixed in place and may be migrated between physical hosts and storage devices while the system is running.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">StoragePools aggregate a collection of DataStores and define the replication or erasure-coding strategy that is used for objects allocated within the pool.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Objects are allocated from StoragePools and use the replication\/erasure-coding strategy defined for the pool they are allocated from. Each object has an ObjectRevision that can be used for ensuring transaction consistency as well as a ReferenceCount. Objects are deleted when a transaction sets their ReferenceCount to zero.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Much like C\/C++ programs, you get a small, binary pointer when you allocate an object in Aspen and you embed those pointers within other objects to create distributed data structures.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Movable DataStores, StoragePools, Transactional Objects, ObjectPointers, and Distributed Data Structures are the foundational concepts in Aspen. There&#8217;s a lot more to the system, of course, but those are the core and a few things fall out of it that might not be immediately obvious.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The first is that, because DataStores may be migrated between machines and physical storage media, you can create distributed data structures that take advantage of heterogeneous storage media and geo-location. If you want a distributed B-Tree for fast lookups of bulk data, place the upper tiers of the tree on co-located machines with NVMe media and the large data on geo-dispersed traditional spinning hard drives. If you have a distributed file system where content is organized by region, ensure that the \/root\/region\/sacramento directory content and files are physically located there while the rest of your data is elsewhere. If the office relocates, rename the folder to \/root\/region\/dallas and migrate the underlying DataStores to Texas. The degree of flexibility is unparalleled and it opens up options for application designs that aren&#8217;t practical in any other major system.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A second major advantage is that Aspen makes it easy to define crash-proof durable tasks that provide exactly-once guarantees. To do this, we use a UUID that defines the task type and a state object that contains the crash-proof state needed to carry out the task. Each time a step in the task is completed, the state is updated to point to the next step in the process. Should the host running the durable task crash, another host can resume the task by using the UUID and state object. The resumed task will simply pick up where the crash happend and restart te last operation. When an exactly-once operation is needed, add the objects being modified to the transaction that updates the task state to the next step. It&#8217;s as simple as that. Aspen is a self-hosting system and makes use of this strategy to implement many of its internal features.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading has-background-color has-text-color has-link-color wp-elements-5\">More information<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/github.com\/aspen-ddp\/aspen#more-information\"><\/a>A full description of how Aspen works and its design trade-offs may be found in the <a href=\"https:\/\/aspen-ddp.org\/staging\/5074\/aspen-architecture\/\" data-type=\"page\" data-id=\"44\">Architecture<\/a> section.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading has-background-color has-text-color has-link-color wp-elements-6\">Git Repsitory<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Aspen\u2019s source code may be found on <a href=\"https:\/\/github.com\/aspen-ddp\/aspen\">GitHub<\/a> in a <a href=\"https:\/\/github.com\/aspen-ddp\/aspen\" data-type=\"link\" data-id=\"https:\/\/github.com\/aspen-ddp\/aspen\">project repository<\/a> and is distributed under the LGPL license. The intent behind this license is to encourage its use in both open-source and proprietary projects with the main ask being that any enhancements made to Aspen itself in support of an application be contributed back to the community so that other projects may benefit from the improvements.\u00a0<\/p>\n\n\n\n<div style=\"height:48px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/github.com\/aspen-ddp TL;DR Motivation Aspen solves a problem you probably didn&#8217;t realize we have and that&#8217;s the lack of a general-purpose distributed data platform for building various kinds of distributed systems. To date, if you want to build a distributed system, you essentially have two choices. You either start from scratch or you use composition. In [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"nf_dc_page":"home","om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"footnotes":""},"class_list":["post-23","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Home - Aspen Distributed Data Platform<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/aspen-ddp.org\/staging\/5074\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Home - Aspen Distributed Data Platform\" \/>\n<meta property=\"og:description\" content=\"https:\/\/github.com\/aspen-ddp TL;DR Motivation Aspen solves a problem you probably didn&#8217;t realize we have and that&#8217;s the lack of a general-purpose distributed data platform for building various kinds of distributed systems. To date, if you want to build a distributed system, you essentially have two choices. You either start from scratch or you use composition. In [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/aspen-ddp.org\/staging\/5074\/\" \/>\n<meta property=\"og:site_name\" content=\"Aspen Distributed Data Platform\" \/>\n<meta property=\"article:modified_time\" content=\"2026-09-13T23:27:32+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/aspen-ddp.org\\\/staging\\\/5074\\\/\",\"url\":\"https:\\\/\\\/aspen-ddp.org\\\/staging\\\/5074\\\/\",\"name\":\"Home - Aspen Distributed Data Platform\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/xxv.owo.mybluehost.me\\\/#website\"},\"datePublished\":\"2024-11-26T21:14:25+00:00\",\"dateModified\":\"2026-09-13T23:27:32+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/aspen-ddp.org\\\/staging\\\/5074\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/aspen-ddp.org\\\/staging\\\/5074\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/aspen-ddp.org\\\/staging\\\/5074\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/aspen-ddp.org\\\/staging\\\/5074\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Home\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/xxv.owo.mybluehost.me\\\/#website\",\"url\":\"https:\\\/\\\/xxv.owo.mybluehost.me\\\/\",\"name\":\"Aspen Distributed Data Platform\",\"description\":\"A scalable and general-purpose distributed data platform for building higher-level distributed systems\",\"publisher\":{\"@id\":\"https:\\\/\\\/xxv.owo.mybluehost.me\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/xxv.owo.mybluehost.me\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/xxv.owo.mybluehost.me\\\/#organization\",\"name\":\"Aspen Distributed Data Platform\",\"url\":\"https:\\\/\\\/xxv.owo.mybluehost.me\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/xxv.owo.mybluehost.me\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/aspen-ddp.org\\\/staging\\\/5074\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/aspen-logo.png\",\"contentUrl\":\"https:\\\/\\\/aspen-ddp.org\\\/staging\\\/5074\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/aspen-logo.png\",\"width\":609,\"height\":608,\"caption\":\"Aspen Distributed Data Platform\"},\"image\":{\"@id\":\"https:\\\/\\\/xxv.owo.mybluehost.me\\\/#\\\/schema\\\/logo\\\/image\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Home - Aspen Distributed Data Platform","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/aspen-ddp.org\/staging\/5074\/","og_locale":"en_US","og_type":"article","og_title":"Home - Aspen Distributed Data Platform","og_description":"https:\/\/github.com\/aspen-ddp TL;DR Motivation Aspen solves a problem you probably didn&#8217;t realize we have and that&#8217;s the lack of a general-purpose distributed data platform for building various kinds of distributed systems. To date, if you want to build a distributed system, you essentially have two choices. You either start from scratch or you use composition. In [&hellip;]","og_url":"https:\/\/aspen-ddp.org\/staging\/5074\/","og_site_name":"Aspen Distributed Data Platform","article_modified_time":"2026-09-13T23:27:32+00:00","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/aspen-ddp.org\/staging\/5074\/","url":"https:\/\/aspen-ddp.org\/staging\/5074\/","name":"Home - Aspen Distributed Data Platform","isPartOf":{"@id":"https:\/\/xxv.owo.mybluehost.me\/#website"},"datePublished":"2024-11-26T21:14:25+00:00","dateModified":"2026-09-13T23:27:32+00:00","breadcrumb":{"@id":"https:\/\/aspen-ddp.org\/staging\/5074\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/aspen-ddp.org\/staging\/5074\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/aspen-ddp.org\/staging\/5074\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/aspen-ddp.org\/staging\/5074\/"},{"@type":"ListItem","position":2,"name":"Home"}]},{"@type":"WebSite","@id":"https:\/\/xxv.owo.mybluehost.me\/#website","url":"https:\/\/xxv.owo.mybluehost.me\/","name":"Aspen Distributed Data Platform","description":"A scalable and general-purpose distributed data platform for building higher-level distributed systems","publisher":{"@id":"https:\/\/xxv.owo.mybluehost.me\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/xxv.owo.mybluehost.me\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/xxv.owo.mybluehost.me\/#organization","name":"Aspen Distributed Data Platform","url":"https:\/\/xxv.owo.mybluehost.me\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/xxv.owo.mybluehost.me\/#\/schema\/logo\/image\/","url":"https:\/\/aspen-ddp.org\/staging\/5074\/wp-content\/uploads\/2026\/09\/aspen-logo.png","contentUrl":"https:\/\/aspen-ddp.org\/staging\/5074\/wp-content\/uploads\/2026\/09\/aspen-logo.png","width":609,"height":608,"caption":"Aspen Distributed Data Platform"},"image":{"@id":"https:\/\/xxv.owo.mybluehost.me\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/aspen-ddp.org\/staging\/5074\/wp-json\/wp\/v2\/pages\/23","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aspen-ddp.org\/staging\/5074\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/aspen-ddp.org\/staging\/5074\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/aspen-ddp.org\/staging\/5074\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aspen-ddp.org\/staging\/5074\/wp-json\/wp\/v2\/comments?post=23"}],"version-history":[{"count":11,"href":"https:\/\/aspen-ddp.org\/staging\/5074\/wp-json\/wp\/v2\/pages\/23\/revisions"}],"predecessor-version":[{"id":80,"href":"https:\/\/aspen-ddp.org\/staging\/5074\/wp-json\/wp\/v2\/pages\/23\/revisions\/80"}],"wp:attachment":[{"href":"https:\/\/aspen-ddp.org\/staging\/5074\/wp-json\/wp\/v2\/media?parent=23"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}