LucidLink Connect streams S3 buckets without the ingest headache

Cloud file streamer LucidLink's new Connect product lets users hook up existing data stores, such as AWS S3, to its Filespaces domain, and then stream them in real-time to other LucidLink users without having to copy them into Filespaces first.

LucidLink sees itself as a kind of globally capable data-delivery pump, using either its own or a customer's existing data stores. Consider it digitally analogous to a water supply utility, with pipelines to customers delivering water from its own or a customer's source reservoirs. It currently manages more than 90 petabytes globally for 100,000-plus users across 4,000-plus businesses in 150-plus countries, and has around 230 employees. T

Filespaces streams parts of files in real time from the Filespaces store in the cloud to globally distributed users on desktop or mobile devices, who then access the file data as if it's stored on local drives. This is different from other cloud file services providers such as CTERA, Egnyte, Nasuni, and Panzura, which share whole files with users. 

Now, by using LucidLink APIs, Connect enables external S3 buckets to be integrated into its Filespaces domain, and shipped out to users in the same way. The source file native formats and workflows are unaffected, with LucidLink being a transparent data delivery plumber, as it were. Filespaces integration setup can be automated and repeatable workflows orchestrated.

Peter Thompson, LucidLink founder and CEO, told us: "LucidLink has been streaming files from the cloud at petabyte scale for 10 years. LucidLink Connect brings that proven capability to existing data, making it instantly usable without forcing customers to change where it lives or how their systems work."

The Filespaces cloud store is object-based with LucidLink overlaying its proprietary format on top. Up until now, existing customer data has had to be ingested into Filespaces.

This could be useful for application spaces that involve lots of large files, such as media and entertainment (M&E), where LucidLink is well known, architecture, engineering and construction (AEC), media and digital asset management (MAM and DAM), enterprises and, in the last couple of years, AI training and inference. Groups of local users can transparently share a single level 2 TeamCache  – a pass-through cache  – reducing the network bandwidth needed to stream file data directly to each and every one of them. This TeamCache technology is currently in an early access phase.

Thompson told us, "More and more of our customers would come to us and say, 'Hey, we actually have huge libraries already out there that we would like to activate. We would like to take these petabytes worth of files that we've created or media assets or footage, and we want to access that into a workspace.' So we do the collaboration within the workspace, but we would like to activate these already existing files in object storage without ingesting them, without moving them"

Existing file-based workflows can now have S3 datasets included within them, extending LucidLink's reach. Customers can continue working with files in their original format, structure and location.

The activation is done through LucidLink APIs and effectively by extracting metadata. Chief product officer Richard Yu told us, "We... are given access to the existing S3 libraries and the data that's already there. And our technology... basically allows the user to create LucidLink metadata that then enables us to stream and bring down parts and pieces of those existing libraries as the end user requires them. That's how then we can represent all those native S3 libraries as directories, folders, and files in the same LucidLink representation of a file system."

Customer developers, he said, have "programmatic access to our tools with APIs and software development kits so that we have large customers actually orchestrating and building custom solutions using these APIs and SDKs." (Note that linking via CLI will be enabled in a future release.)

This enables LucidLink to step outwards from its M&E heartland. As Yu explained, "We become much more of a horizontally applicable product platform. We have customers today in other industries that already leveraged LucidLink to manage their large files and large-file collaboration, architecture, engineering, construction, healthcare, very secure datasets and FinServe and government use cases."

In a way, he says, "I think we've kind of come full circle to having that technology, finding initial product-market fit in media and entertainment workflows, and then now becoming much more horizontal, and even being considered and discussed and tested for AI-type workloads."

We recently reported on the competing Suite Studios product, which enables customers to stream byte-range portions of large files stored as S3 objects. The users also get local disk-like performance.

he Connect product is available now on LucidLink Enterprise plans, and purchasing via AWS Marketplace is coming soon. Read a LucidLink Connect technical overview here. S3-compatible stores are supported now. Azure Blob storage support is planned, as is support for other non-object storage types, and also TeamCache.